Nov 03 17:32:57.462430 ip-10-0-131-172 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:32:57.861435 ip-10-0-131-172 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:32:57.861435 ip-10-0-131-172 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:32:57.861435 ip-10-0-131-172 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:32:57.861435 ip-10-0-131-172 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:32:57.865856 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865771    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.865856 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865852    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.865856 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865857    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.865856 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865862    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865866    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865871    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865875    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865879    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865883    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865887    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865890    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865894    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865898    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865902    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865907    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865911    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865915    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865918    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865922    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865926    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865930    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865934    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.866125 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865937    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865941    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865945    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865950    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865953    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865958    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865962    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865966    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865970    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865975    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865978    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865983    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865988    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865993    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.865997    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866002    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866006    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866011    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866015    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866020    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.866577 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866024    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866028    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866032    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866041    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866048    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866054    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866059    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866064    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866069    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866073    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866077    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866082    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866742    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866753    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866759    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866764    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866770    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866776    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866780    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.867458 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866785    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866789    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866794    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866800    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866806    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866812    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866816    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866821    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866825    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866829    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866834    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866838    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866842    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866847    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866851    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866856    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866860    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866865    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866870    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.868201 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866875    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866880    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866884    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866889    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866893    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866897    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866901    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866905    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866909    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866913    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866917    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866921    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866926    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866930    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866934    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866938    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866943    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866947    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866951    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866955    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.868701 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866960    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.869188 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866964    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.869188 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866967    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.869188 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866971    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.869188 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866977    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.869188 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.866980    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869234    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869252    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869255    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869258    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869261    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869263    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869268    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869271    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869274    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869276    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869279    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869282    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869284    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869287    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869289    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869292    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869295    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869297    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.869309 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869299    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869302    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869304    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869307    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869311    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869316    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869319    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869322    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869325    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869327    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869330    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869333    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869335    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869344    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869347    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869350    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869353    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869355    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869358    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.869787 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869360    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869363    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869365    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869367    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869370    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869372    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869375    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869377    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869380    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869383    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869385    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869388    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869391    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869394    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869396    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869401    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869404    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869406    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869409    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869412    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.870253 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869415    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869417    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869420    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869422    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869425    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869427    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869430    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869432    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869435    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869437    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869440    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869442    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869445    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869447    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869450    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869452    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869454    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869457    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869459    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.870740 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869462    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869465    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869467    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869470    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869472    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869475    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869478    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869483    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869486    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869489    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869492    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869495    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869498    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869500    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869504    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869506    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869509    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869511    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869514    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.871215 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869516    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869518    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869521    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869523    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869526    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869528    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869531    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869533    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869535    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.869538    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870030    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870035    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870039    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870044    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870048    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870052    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870055    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870058    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.871673 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870060    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870066    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870069    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870072    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870074    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870077    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870080    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870082    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870085    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870087    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870090    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870092    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870094    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870097    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870116    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870118    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870121    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870123    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870126    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870129    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.872142 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870131    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870134    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870136    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870138    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870141    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870144    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870146    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870149    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870152    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870155    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870157    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870160    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870163    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870165    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870168    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870171    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870174    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870176    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870179    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.872621 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870181    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870184    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870186    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870189    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870191    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870194    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870196    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870199    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870201    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870204    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870206    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870209    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870211    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870224    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870227    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870229    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870232    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870234    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870237    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870239    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.873065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870242    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870245    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870248    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870250    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870254    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870256    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870259    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870262    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870265    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870268    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870271    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870273    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870276    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870279    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870282    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870284    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870286    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870289    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870291    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870294    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.873607 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870296    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870299    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870301    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870304    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870306    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870308    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870311    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870313    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870316    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870318    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870321    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870324    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870327    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870329    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870332    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870335    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870338    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870340    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.870342    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871271    2576 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871282    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:32:57.874065 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871290    2576 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871295    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871300    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871303    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871307    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871312    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871315    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871319    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871325    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871328    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871331    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871334    2576 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871337    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871340    2576 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871343    2576 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871346    2576 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871352    2576 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871355    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871358    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871361    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871365    2576 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871368    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871371    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:32:57.874604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871374    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871378    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871381    2576 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871385    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871388    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871391    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871394    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871403    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871406    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871409    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871412    2576 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871415    2576 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871418    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871422    2576 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871425    2576 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871428    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871431    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871436    2576 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871440    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871443    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871446    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871449    2576 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871452    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871455    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871458    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:32:57.875151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871461    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871464    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871466    2576 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871469    2576 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871474    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871477    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871480    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871483    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871486    2576 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871489    2576 flags.go:64] FLAG: --help="false"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871492    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871495    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871498    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871500    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871504    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871507    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871510    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871513    2576 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871516    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871519    2576 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871522    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871525    2576 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871527    2576 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871530    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:32:57.875747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871535    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871538    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871541    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871544    2576 flags.go:64] FLAG: --lock-file=""
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871546    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871549    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871552    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871558    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871561    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871564    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871567    2576 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871570    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871573    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871576    2576 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871579    2576 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871583    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871586    2576 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871590    2576 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871593    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871596    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871599    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871602    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871605    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871608    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871611    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:32:57.876347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871619    2576 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871622    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871625    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871628    2576 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871631    2576 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871634    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871637    2576 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871641    2576 flags.go:64] FLAG: --port="10250"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871644    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871647    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-04294316c535a5d0e"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871651    2576 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871653    2576 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871656    2576 flags.go:64] FLAG: --register-node="true"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871659    2576 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871663    2576 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871666    2576 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871669    2576 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871671    2576 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871675    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871678    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871682    2576 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871684    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871687    2576 flags.go:64] FLAG: --runonce="false"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871691    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871694    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871696    2576 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:32:57.876969 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871699    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871702    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871705    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871708    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871711    2576 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871714    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871717    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871720    2576 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871722    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871726    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871729    2576 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871731    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871737    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871740    2576 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871743    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871748    2576 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871751    2576 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871754    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871757    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871760    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871763    2576 flags.go:64] FLAG: --v="2"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871767    2576 flags.go:64] FLAG: --version="false"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871771    2576 flags.go:64] FLAG: --vmodule=""
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871775    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.871778    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:32:57.877681 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871909    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871913    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871916    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871919    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871921    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871924    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871926    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871929    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871932    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871934    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871936    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871942    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871945    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871947    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871950    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871952    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871955    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871958    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871960    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871963    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.878341 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871967    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871969    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871972    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871974    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871977    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871979    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871982    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871984    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871987    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871989    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871992    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871994    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871997    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.871999    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872002    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872004    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872007    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872009    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872011    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.878824 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872014    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872016    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872019    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872021    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872026    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872029    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872032    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872034    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872036    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872039    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872041    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872045    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872047    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872051    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872054    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872056    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872059    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872061    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872064    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872066    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.879287 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872069    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872071    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872074    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872076    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872079    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872082    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872084    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872087    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872089    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872092    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872095    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872097    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872113    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872116    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872118    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872120    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872124    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872127    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872129    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872132    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.879761 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872135    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872137    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872140    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872143    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872146    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872150    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872152    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872155    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872158    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872160    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872164    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872168    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872172    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872176    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872180    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872183    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872186    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872188    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872191    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.880248 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872193    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872196    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872198    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872200    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872203    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872205    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872208    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872210    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.872237    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872276    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872281    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872284    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872287    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872290    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872292    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872295    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.880704 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872298    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872301    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872304    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872306    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872309    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872312    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872314    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872317    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872319    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872322    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872324    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872327    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872329    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872332    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872334    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872336    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872339    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872341    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872344    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.881074 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872348    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872352    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872355    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872359    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872362    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872366    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872369    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872372    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872375    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872377    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872380    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872382    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872385    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872387    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872390    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872392    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872396    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872398    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872400    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.881570 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872403    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872405    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872408    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872410    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872413    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872415    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872418    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872420    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872423    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872426    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872429    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872431    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872433    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872436    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872438    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872441    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872443    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872446    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872448    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872451    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.882020 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872455    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872457    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872460    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872462    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872465    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872467    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872469    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872472    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872474    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872477    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872480    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872482    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872485    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872487    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872490    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872492    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872495    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872497    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872499    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.882501 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872502    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872504    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872507    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872509    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872512    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872514    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872517    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872519    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872521    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872524    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872526    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872528    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872531    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872535    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872537    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872540    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872542    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872545    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872547    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872550    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.882933 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872554    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.872558    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.879889    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.879906    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.880658    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.880666    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880722    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880727    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880730    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880734    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880737    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880740    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880742    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880745    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880748    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880750    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880753    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880756    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880759    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.883405 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880761    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880764    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880766    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880769    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880771    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880774    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880776    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880779    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880781    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880784    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880786    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880789    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880792    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880794    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880797    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880800    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880803    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880806    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880809    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880811    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.883903 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880814    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880816    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880819    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880821    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880824    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880826    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880829    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880831    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880834    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880837    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880839    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880841    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880844    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880847    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880849    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880852    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880854    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880857    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880860    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.884380 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880865    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880868    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880871    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880875    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880878    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880881    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880884    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880887    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880890    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880893    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880896    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880898    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880901    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880904    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880906    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880908    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880911    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880913    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880916    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880918    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.884874 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880921    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880923    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880927    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880930    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880934    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880936    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880939    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880941    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880944    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880946    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880949    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880951    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880953    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880956    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880958    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880961    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880964    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880966    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880969    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.885364 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880972    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880974    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880977    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880980    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880983    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880986    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880988    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880991    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880993    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880996    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.880998    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881001    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881003    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881005    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881008    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.881035    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:32:57.885919 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881192    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881197    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881201    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881204    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881206    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881209    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881212    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881214    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881217    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881219    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881222    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881224    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881227    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881230    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881232    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881235    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881238    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881240    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881243    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.886409 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881247    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881251    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881255    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881258    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881261    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881263    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881266    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881269    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881271    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881273    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881276    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881278    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881281    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881283    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881285    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881288    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881290    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881293    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881295    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881298    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.887197 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881301    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881303    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881305    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881308    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881310    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881313    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881316    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881318    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881320    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881323    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881325    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881328    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881330    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881333    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881335    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881338    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881341    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881344    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881346    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.887828 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881348    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881351    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881353    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881356    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881358    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881361    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881363    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881365    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881367    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881370    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881374    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881378    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881381    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881384    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881386    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881389    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881391    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881394    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881396    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.888337 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881399    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881401    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881404    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881406    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881408    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881411    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881413    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881416    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881418    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881421    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881423    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881426    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881428    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881431    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881433    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881436    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881438    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881441    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881444    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881446    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.888806 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881448    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881451    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881453    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881456    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881458    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881460    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881463    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881466    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881468    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.881491    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881522    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881526    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881529    2576 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881532    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881535    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881538    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:32:57.889306 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881540    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881544    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881546    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881549    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881552    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881554    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881557    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881560    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881562    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881565    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881567    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881570    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881572    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881575    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881577    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881580    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881583    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881586    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881588    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:32:57.889835 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881591    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881593    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881595    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881598    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881601    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881603    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881605    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881608    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881610    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881613    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881615    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881618    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881621    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881625    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881629    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881633    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881635    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881638    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881641    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:32:57.890604 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881643    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881646    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881648    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881651    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881654    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881657    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881659    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881661    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881664    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881666    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881668    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881671    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881673    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881675    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881678    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881680    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881683    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881685    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881688    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881690    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:32:57.891302 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881693    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881695    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881697    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881700    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881702    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881704    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881707    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881709    2576 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881712    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881714    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881716    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881719    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881721    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881724    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881726    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881739    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881745    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881747    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881750    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881752    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:32:57.891846 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881754    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881757    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881759    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881762    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881764    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881766    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881769    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881771    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881774    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881776    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881780    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881783    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881787    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881789    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881792    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881794    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881797    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881799    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881801    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:32:57.892311 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881804    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881806    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:57.881808    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.881863    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.884296    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.885305    2576 server.go:1014] "Starting client certificate rotation"
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.885418    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.885455    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.887460    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:32:57.892779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.888625    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:32:57.901018 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.901001    2576 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:32:57.905023 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.905002    2576 log.go:25] "Validated CRI v1 image API"
Nov 03 17:32:57.905878 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.905855    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:32:57.908464 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.908444    2576 fs.go:135] Filesystem UUIDs: map[1cab9925-e0a1-4854-b491-2b24c98b6fb2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7a419278-b02a-4d76-b90d-b2e8e3d9846e:/dev/nvme0n1p4]
Nov 03 17:32:57.908522 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.908463    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 17:32:57.914568 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.914465    2576 manager.go:217] Machine: {Timestamp:2026-11-03 17:32:57.912979361 +0000 UTC m=+0.350205429 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093897 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d2d378e1e48920c2567d358c4561 SystemUUID:ec27d2d3-78e1-e489-20c2-567d358c4561 BootID:0c63f996-05c6-47c2-bd3e-6b62216945a3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:cf:1a:9d:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:cf:1a:9d:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:dd:ac:62:7d:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:32:57.914568 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.914568    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:32:57.914677 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.914644    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:32:57.917930 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.917910    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:32:57.918058 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.917932    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-172.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:32:57.918118 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.918061    2576 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:32:57.918118 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.918069    2576 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:32:57.918174 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.918123    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:32:57.918174 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.918143    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:32:57.919396 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.919386    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:32:57.923626 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.923614    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:32:57.923748 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.923739    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:32:57.924557 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.924547    2576 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:32:57.925252 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.924561    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:32:57.925252 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.924571    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:32:57.925252 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.924581    2576 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:32:57.925252 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.924589    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:32:57.925252 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.924909    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:32:57.927305 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.927278    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:32:57.934692 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.934677    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:32:57.935681 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935666    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:32:57.935747 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935708    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:32:57.935898 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935885    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:32:57.935956 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935906    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:32:57.935956 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935916    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:32:57.935956 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935925    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:32:57.935956 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935934    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:32:57.935956 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.935943    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:32:57.936617 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.936606    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:32:57.936665 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.936621    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:32:57.936665 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.936632    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:32:57.936665 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.936642    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:32:57.936665 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.936655    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:32:57.936807 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.936669    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:32:57.937425 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.937414    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:32:57.937475 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.937430    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:32:57.942148 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.942128    2576 server.go:1263] "Started kubelet"
Nov 03 17:32:57.942233 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.942198    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:32:57.942301 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.942231    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:32:57.942350 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.942309    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:32:57.942938 ip-10-0-131-172 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:32:57.943721 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.943705    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:32:57.945284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.945271    2576 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:32:57.947650 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:57.947633    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 17:32:57.950613 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.950591    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:32:57.951169 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.951152    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:32:57.951325 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:57.951294    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:57.951977 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.951959    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:32:57.951977 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.951961    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:32:57.952126 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.951987    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:32:57.952126 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952077    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:32:57.952126 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952088    2576 factory.go:55] Registering systemd factory
Nov 03 17:32:57.952126 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952094    2576 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:32:57.952292 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952134    2576 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:32:57.952292 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952144    2576 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:32:57.952886 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952867    2576 factory.go:153] Registering CRI-O factory
Nov 03 17:32:57.952886 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952886    2576 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:32:57.953036 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952903    2576 factory.go:103] Registering Raw factory
Nov 03 17:32:57.953036 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.952916    2576 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:32:57.954148 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.954134    2576 manager.go:319] Starting recovery of all containers
Nov 03 17:32:57.965261 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.965239    2576 manager.go:324] Recovery completed
Nov 03 17:32:57.970145 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.970126    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:57.972009 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.971984    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:57.972147 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.972028    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:57.972147 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.972042    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:57.972591 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.972577    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:57.972672 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.972579    2576 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:32:57.972672 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.972627    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:32:57.972672 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.972648    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:32:57.973254 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.973241    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:57.973299 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.973268    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:57.973299 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.973277    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:57.974749 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.974736    2576 policy_none.go:50] "Start"
Nov 03 17:32:57.974796 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.974751    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:32:57.974796 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.974764    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:32:57.976677 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:57.976667    2576 policy_none.go:44] "Start"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.022728    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.027509    2576 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.027552    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.027564    2576 server.go:87] "Starting device plugin registration server"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.027804    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.027817    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.028243    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.028317    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.028326    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.029057    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.029182    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.037489    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.038784    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.038800    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.038816    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:32:58.041501 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.038844    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 17:32:58.086205 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.086172    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-172.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 17:32:58.086205 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.086190    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-172.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 17:32:58.128573 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.128516    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:58.129420 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.129404    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:58.129497 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.129433    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:58.129497 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.129445    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:58.129497 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.129478    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.136196 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.135220    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.us-west-1.compute.internal.18e4995d61336493  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 17:32:57.942066323 +0000 UTC m=+0.379292391,LastTimestamp:2026-11-03 17:32:57.942066323 +0000 UTC m=+0.379292391,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.us-west-1.compute.internal,}"
Nov 03 17:32:58.139772 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.139747    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"]
Nov 03 17:32:58.139840 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.139813    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:58.140512 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.140496    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:58.140578 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.140527    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:58.140578 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.140537    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:58.142679 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.142666    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:58.142823 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.142809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.142900 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.142839    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:58.143404 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.143385    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:58.143497 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.143418    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:58.143497 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.143428    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:58.143497 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.143385    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:58.143497 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.143492    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:58.143660 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.143514    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:58.145538 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.145523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.145596 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.145552    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:58.146297 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.146282    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:58.146357 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.146315    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:58.146357 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.146331    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:58.167364 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.167346    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.169280 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.169263    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.210549 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.210520    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:32:58.210670 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.210569    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sgmgb"
Nov 03 17:32:58.250238 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.250200    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-172.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 17:32:58.250380 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.250254    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 17:32:58.250380 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.250188    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-172.us-west-1.compute.internal.18e4995d62fc4986  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-172.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 17:32:57.97200935 +0000 UTC m=+0.409235429,LastTimestamp:2026-11-03 17:32:57.97200935 +0000 UTC m=+0.409235429,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.us-west-1.compute.internal,}"
Nov 03 17:32:58.250380 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.250305    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 17:32:58.253052 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.253023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.253162 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.253070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.253162 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.253118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3eefabd2e362fe1b3609803d32f993c7-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"3eefabd2e362fe1b3609803d32f993c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.353946 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.353911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.353946 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.353946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3eefabd2e362fe1b3609803d32f993c7-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"3eefabd2e362fe1b3609803d32f993c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.354220 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.353963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.354220 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.353996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.354220 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.354018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3eefabd2e362fe1b3609803d32f993c7-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"3eefabd2e362fe1b3609803d32f993c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.354220 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.354017    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.409294 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.409225    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:32:58.411146 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.411125    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-172.us-west-1.compute.internal\" not found" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.463824 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.463800    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-172.us-west-1.compute.internal\" not found" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.472852 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.472835    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.474996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.474980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.611465 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.611427    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:58.614836 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.614810    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:58.614958 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.614845    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:58.614958 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.614859    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:58.614958 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.614892    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.677831 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.677744    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:58.677831 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.677779    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-172.us-west-1.compute.internal\": node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:58.794474 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.794436    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:58.884955 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.884922    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 17:32:58.885512 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:58.885145    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 17:32:58.885512 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.885123    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa49546cbf5c9454088617b8e9f7adf8-66509c371b41a33e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-172.us-west-1.compute.internal.18e4995d62fcabc5\": read tcp 10.0.131.172:51586->3.149.82.99:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-172.us-west-1.compute.internal.18e4995d62fcabc5  default   4786 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-172.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-172.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-172.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 17:32:57 +0000 UTC,LastTimestamp:2026-11-03 17:32:58.129437656 +0000 UTC m=+0.566663724,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-172.us-west-1.compute.internal,}"
Nov 03 17:32:58.895209 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.895184    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:58.995956 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:58.995866    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:59.096586 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:59.096561    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:59.197475 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:59.197447    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:59.207598 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.207586    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:32:59.208556 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.208538    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:32:59.208653 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.208568    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:32:59.208653 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.208580    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:32:59.216730 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.216698    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:27:59 +0000 UTC" deadline="2028-05-30 06:05:16.957609402 +0000 UTC"
Nov 03 17:32:59.216730 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.216727    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13764h32m17.740887599s"
Nov 03 17:32:59.245071 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.245048    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:32:59.298154 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:32:59.298069    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:32:59.356450 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.356412    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:32:59.367451 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.367426    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:32:59.451546 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.451509    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:59.510260 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:59.510231    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe46139cd173525f608561351491180.slice/crio-d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439 WatchSource:0}: Error finding container d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439: Status 404 returned error can't find the container with id d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439
Nov 03 17:32:59.511372 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.511357    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:32:59.512347 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.512332    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:32:59.520127 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:32:59.520091    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:32:59.527268 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:32:59.527243    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eefabd2e362fe1b3609803d32f993c7.slice/crio-8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4 WatchSource:0}: Error finding container 8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4: Status 404 returned error can't find the container with id 8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4
Nov 03 17:33:01.238433 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.238419    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:33:01.590201 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.590129    2576 apiserver.go:52] "Watching apiserver"
Nov 03 17:33:01.617311 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.617291    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 17:33:01.702793 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.702764    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:33:01.703089 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.703068    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hcf4k","openshift-multus/network-metrics-daemon-2ccdq","openshift-network-diagnostics/network-check-target-7v8km","openshift-network-operator/iptables-alerter-88fkm","openshift-cluster-node-tuning-operator/tuned-vcxvs","openshift-multus/multus-7qrt7","openshift-multus/multus-additional-cni-plugins-vfd7n","kube-system/konnectivity-agent-t4sns","kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal","openshift-image-registry/node-ca-9n9mf","kube-system/global-pull-secret-syncer-648bg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-rgmgf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"]
Nov 03 17:33:01.706780 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.706759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.709059 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.709039    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.711658 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.711639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:01.711721 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.711703    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:01.713970 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.713923    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.716176 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.716160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.721306 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.719130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.721851 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.721830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.724049 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.724031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.726435 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726417    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:01.726521 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.726498    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:01.726604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.726660 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.726660 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5km7g\" (UniqueName: \"kubernetes.io/projected/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-kube-api-access-5km7g\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.726660 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-socket-dir-parent\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.726809 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-netns\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.726809 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-env-overrides\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.726809 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-tuned\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.726809 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-kubelet\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-script-lib\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysconfig\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-systemd\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-run\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-multus-certs\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.726979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-system-cni-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-cnibin\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-var-lib-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727037 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-netd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-kubernetes\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-k8s-cni-cncf-io\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-etc-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-config\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-iptables-alerter-script\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-var-lib-kubelet\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-cni-binary-copy\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-conf-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-node-log\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727273    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-conf\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-bin\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-netns\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.727477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-host-slash\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-lib-modules\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727470    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-multus\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-hostroot\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtkn\" (UniqueName: \"kubernetes.io/projected/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-kube-api-access-lbtkn\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-system-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-cnibin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-daemon-config\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-etc-kubernetes\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-slash\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-ovn\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-tmp\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-modprobe-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.728235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kmt\" (UniqueName: \"kubernetes.io/projected/8222297a-ec44-495a-b6bd-6b58501aadd4-kube-api-access-44kmt\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-kubelet\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-log-socket\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-bin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-systemd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-host\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-systemd-units\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.727995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-sys\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.728008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9pr\" (UniqueName: \"kubernetes.io/projected/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-kube-api-access-pz9pr\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.728023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2th75\" (UniqueName: \"kubernetes.io/projected/01b8876a-2097-40ee-9b09-4ec46606a7b1-kube-api-access-2th75\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.729051 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.728619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.730815 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.730797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.730914 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.730843    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:01.733095 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.733071    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:01.735255 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.735221    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"3eefabd2e362fe1b3609803d32f993c7","Type":"ContainerStarted","Data":"8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4"}
Nov 03 17:33:01.735328 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.735265    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"dbe46139cd173525f608561351491180","Type":"ContainerStarted","Data":"d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439"}
Nov 03 17:33:01.762389 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.762261    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:33:01.762389 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.762261    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:33:01.762389 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.762290    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:33:01.762389 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.762304    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:33:01.762389 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.762322    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xxxqk\""
Nov 03 17:33:01.762389 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.762360    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:33:01.765666 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.765653    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:33:01.811257 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.811231    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 17:33:01.811369 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.811294    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k5rmh"
Nov 03 17:33:01.818174 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.818153    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:33:01.828590 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.828642 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-dbus\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.828642 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-host\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.828642 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-systemd-units\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.828772 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c073539-dddd-4fd2-88ee-52fc61b9f50d-hosts-file\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.828772 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:01.828772 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-sys\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.828772 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828736    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-systemd-units\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.828772 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-host\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.828772 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9pr\" (UniqueName: \"kubernetes.io/projected/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-kube-api-access-pz9pr\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.828980 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-sys\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.828980 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2th75\" (UniqueName: \"kubernetes.io/projected/01b8876a-2097-40ee-9b09-4ec46606a7b1-kube-api-access-2th75\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.828980 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.829074 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.828989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829074 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km7g\" (UniqueName: \"kubernetes.io/projected/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-kube-api-access-5km7g\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829074 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829031    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829074 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-socket-dir-parent\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829205 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-netns\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829205 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-env-overrides\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829205 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-socket-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.829205 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-netns\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829205 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-agent-certs\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-socket-dir-parent\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-tuned\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-kubelet\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-script-lib\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829279    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-kubelet\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysconfig\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-systemd\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysconfig\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-run\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-systemd\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829399 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-multus-certs\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-system-cni-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-run\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-cnibin\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-var-lib-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829465    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-system-cni-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-netd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829505    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-var-lib-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-kubernetes\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829519    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829527    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-netd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-k8s-cni-cncf-io\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829543    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-kubernetes\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-etc-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-k8s-cni-cncf-io\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-config\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.829967 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829587    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-etc-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-iptables-alerter-script\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-var-lib-kubelet\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-cni-binary-copy\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-conf-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829694    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-var-lib-kubelet\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-node-log\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-conf-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-device-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829785    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-node-log\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-multus-certs\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-conf\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-bin\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.830609 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-konnectivity-ca\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-cnibin\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829978    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-bin\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.829996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-conf\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-netns\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830093    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-cni-binary-copy\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-host-slash\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830125    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-netns\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-host-slash\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnh6\" (UniqueName: \"kubernetes.io/projected/2c073539-dddd-4fd2-88ee-52fc61b9f50d-kube-api-access-2pnh6\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-lib-modules\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-multus\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-hostroot\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830264    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-lib-modules\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-registration-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.831284 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830279    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830285    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-multus\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtkn\" (UniqueName: \"kubernetes.io/projected/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-kube-api-access-lbtkn\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830316    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-hostroot\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3138bcb-bcf2-49bc-959d-c2177605f704-serviceca\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-system-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-system-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-cnibin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-daemon-config\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-etc-kubernetes\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-etc-kubernetes\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-slash\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830630    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-cnibin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-slash\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-ovn\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-sys-fs\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830712    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-ovn\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832091 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kb4\" (UniqueName: \"kubernetes.io/projected/1e373a48-c67d-4cd3-8837-166a41802085-kube-api-access-p2kb4\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3138bcb-bcf2-49bc-959d-c2177605f704-host\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-tmp\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vjp\" (UniqueName: \"kubernetes.io/projected/d3138bcb-bcf2-49bc-959d-c2177605f704-kube-api-access-g4vjp\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.830979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-kubelet-config\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831353    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-modprobe-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831037    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-modprobe-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831416    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kmt\" (UniqueName: \"kubernetes.io/projected/8222297a-ec44-495a-b6bd-6b58501aadd4-kube-api-access-44kmt\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-kubelet\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-log-socket\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.832814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c073539-dddd-4fd2-88ee-52fc61b9f50d-tmp-dir\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdj6s\" (UniqueName: \"kubernetes.io/projected/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-kube-api-access-sdj6s\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-bin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-systemd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-systemd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.832055    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-log-socket\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.832110    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.832138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-bin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.831770    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-kubelet\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.833564 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.832605    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.834642 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.834617    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.834733 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.834636    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-tuned\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.835955 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.835938    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-tmp\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.863245 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863184    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:33:01.863245 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863234    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:33:01.863357 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863241    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:33:01.863357 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863283    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:33:01.863357 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863308    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:33:01.863511 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863495    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:01.863571 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863533    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:33:01.863571 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863545    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:33:01.863649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863576    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:33:01.863649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863586    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:33:01.863649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863588    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:33:01.863649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863604    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nx9c7\""
Nov 03 17:33:01.863779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863673    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:33:01.863779 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863694    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:33:01.863845 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863796    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:33:01.863845 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.863787    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:33:01.870388 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.870368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-env-overrides\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.870457 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.870424    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-config\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.871569 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.871551    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-daemon-config\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.873294 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.873274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.882744 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.882728    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:33:01.883001 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.882984    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:01.883067 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.883003    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tvvh\""
Nov 03 17:33:01.883067 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.883012    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:33:01.913708 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.913686    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2vfjt\""
Nov 03 17:33:01.913827 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.913814    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2qfzl\""
Nov 03 17:33:01.913906 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.913847    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxb7s\""
Nov 03 17:33:01.913906 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.913856    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:33:01.913906 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.913884    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jxmj\""
Nov 03 17:33:01.914117 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.914100    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xhj6\""
Nov 03 17:33:01.914117 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.914106    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:33:01.914186 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.914124    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfxsl\""
Nov 03 17:33:01.914186 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.914141    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:33:01.914186 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.914148    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:01.914498 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.914479    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:01.914574 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.914503    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:01.914574 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.914517    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v5czc for pod openshift-network-diagnostics/network-check-target-7v8km: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:01.914659 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.914595    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc podName:44f3aa72-92ab-4dff-9c6a-2c78626e8b50 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:02.414574649 +0000 UTC m=+3.187030401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v5czc" (UniqueName: "kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc") pod "network-check-target-7v8km" (UID: "44f3aa72-92ab-4dff-9c6a-2c78626e8b50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:01.916841 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.916820    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2th75\" (UniqueName: \"kubernetes.io/projected/01b8876a-2097-40ee-9b09-4ec46606a7b1-kube-api-access-2th75\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:01.916944 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.916928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtkn\" (UniqueName: \"kubernetes.io/projected/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-kube-api-access-lbtkn\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.917009 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.916928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9pr\" (UniqueName: \"kubernetes.io/projected/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-kube-api-access-pz9pr\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:01.917173 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.917156    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kmt\" (UniqueName: \"kubernetes.io/projected/8222297a-ec44-495a-b6bd-6b58501aadd4-kube-api-access-44kmt\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:01.917262 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.917245    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km7g\" (UniqueName: \"kubernetes.io/projected/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-kube-api-access-5km7g\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.920426 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.920406    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-iptables-alerter-script\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:01.920505 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.920489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-script-lib\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:01.932539 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vjp\" (UniqueName: \"kubernetes.io/projected/d3138bcb-bcf2-49bc-959d-c2177605f704-kube-api-access-g4vjp\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.932611 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-kubelet-config\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.932611 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c073539-dddd-4fd2-88ee-52fc61b9f50d-tmp-dir\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.932611 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj6s\" (UniqueName: \"kubernetes.io/projected/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-kube-api-access-sdj6s\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:01.932764 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.932764 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-dbus\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.932764 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c073539-dddd-4fd2-88ee-52fc61b9f50d-hosts-file\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.932764 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:01.932764 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932729    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-kubelet-config\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.932764 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.932764 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932738    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c073539-dddd-4fd2-88ee-52fc61b9f50d-hosts-file\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-socket-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-dbus\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-agent-certs\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.932813    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932848    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c073539-dddd-4fd2-88ee-52fc61b9f50d-tmp-dir\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.932861    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:02.432846314 +0000 UTC m=+3.205302066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-device-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932930    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-socket-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-konnectivity-ca\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.932980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-device-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnh6\" (UniqueName: \"kubernetes.io/projected/2c073539-dddd-4fd2-88ee-52fc61b9f50d-kube-api-access-2pnh6\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-registration-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933031    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.933044    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:01.933132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3138bcb-bcf2-49bc-959d-c2177605f704-serviceca\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:01.933088    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret podName:9b82e04a-d3d2-43de-ad90-abccdeb48e7e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:02.433073469 +0000 UTC m=+3.205529216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret") pod "global-pull-secret-syncer-648bg" (UID: "9b82e04a-d3d2-43de-ad90-abccdeb48e7e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933086    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-registration-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-sys-fs\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kb4\" (UniqueName: \"kubernetes.io/projected/1e373a48-c67d-4cd3-8837-166a41802085-kube-api-access-p2kb4\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3138bcb-bcf2-49bc-959d-c2177605f704-host\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933221    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3138bcb-bcf2-49bc-959d-c2177605f704-host\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933217    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-sys-fs\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933480    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3138bcb-bcf2-49bc-959d-c2177605f704-serviceca\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:01.933996 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.933710    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-konnectivity-ca\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:01.934976 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.934960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-agent-certs\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:01.997278 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.997250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kb4\" (UniqueName: \"kubernetes.io/projected/1e373a48-c67d-4cd3-8837-166a41802085-kube-api-access-p2kb4\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:01.997403 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.997299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnh6\" (UniqueName: \"kubernetes.io/projected/2c073539-dddd-4fd2-88ee-52fc61b9f50d-kube-api-access-2pnh6\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:01.997403 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:01.997308    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdj6s\" (UniqueName: \"kubernetes.io/projected/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-kube-api-access-sdj6s\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:02.002306 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.002290    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vjp\" (UniqueName: \"kubernetes.io/projected/d3138bcb-bcf2-49bc-959d-c2177605f704-kube-api-access-g4vjp\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:02.036804 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.036779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:02.038945 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.038927    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:02.045026 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.045006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:02.045126 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.045047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:02.047848 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.047822    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8222297a_ec44_495a_b6bd_6b58501aadd4.slice/crio-f396f99503ac8aef8cf922b88db3e418127442b9ccd4a9ec0d5d0e67ad871675 WatchSource:0}: Error finding container f396f99503ac8aef8cf922b88db3e418127442b9ccd4a9ec0d5d0e67ad871675: Status 404 returned error can't find the container with id f396f99503ac8aef8cf922b88db3e418127442b9ccd4a9ec0d5d0e67ad871675
Nov 03 17:33:02.057092 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.057066    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cc09b0_ed27_4664_9d61_81d7e3f7ef09.slice/crio-2ada58c712a4d8da4b6f4215f66659e428cf0b9bd8e38cfbe42c8588f2a307eb WatchSource:0}: Error finding container 2ada58c712a4d8da4b6f4215f66659e428cf0b9bd8e38cfbe42c8588f2a307eb: Status 404 returned error can't find the container with id 2ada58c712a4d8da4b6f4215f66659e428cf0b9bd8e38cfbe42c8588f2a307eb
Nov 03 17:33:02.058357 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.058339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:02.058904 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.058800    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b8876a_2097_40ee_9b09_4ec46606a7b1.slice/crio-f06fe48aa384d0f6d9876177748c36405e6e4845ae0ae70efde4e2c92053636f WatchSource:0}: Error finding container f06fe48aa384d0f6d9876177748c36405e6e4845ae0ae70efde4e2c92053636f: Status 404 returned error can't find the container with id f06fe48aa384d0f6d9876177748c36405e6e4845ae0ae70efde4e2c92053636f
Nov 03 17:33:02.060392 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.060300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:02.062585 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.062563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:02.064607 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.064591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:02.066911 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.066891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:02.067462 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.067376    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c073539_dddd_4fd2_88ee_52fc61b9f50d.slice/crio-4bf56a416a86b91ee9ee9c5473367cf25ea649e42797f5486653dbef49f71b82 WatchSource:0}: Error finding container 4bf56a416a86b91ee9ee9c5473367cf25ea649e42797f5486653dbef49f71b82: Status 404 returned error can't find the container with id 4bf56a416a86b91ee9ee9c5473367cf25ea649e42797f5486653dbef49f71b82
Nov 03 17:33:02.070748 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.070724    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c37bc06_03be_48f6_9bd9_f6eb3e711ef0.slice/crio-473ffa27f4d4206455c8e944d4ee676dcf7c1786073575e82db66ead79e62989 WatchSource:0}: Error finding container 473ffa27f4d4206455c8e944d4ee676dcf7c1786073575e82db66ead79e62989: Status 404 returned error can't find the container with id 473ffa27f4d4206455c8e944d4ee676dcf7c1786073575e82db66ead79e62989
Nov 03 17:33:02.075772 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.075746    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e373a48_c67d_4cd3_8837_166a41802085.slice/crio-42e50590bf2ca76e52dcd453d3f6036937af7061297cd967de8296b60d4c772c WatchSource:0}: Error finding container 42e50590bf2ca76e52dcd453d3f6036937af7061297cd967de8296b60d4c772c: Status 404 returned error can't find the container with id 42e50590bf2ca76e52dcd453d3f6036937af7061297cd967de8296b60d4c772c
Nov 03 17:33:02.079741 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.079717    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3138bcb_bcf2_49bc_959d_c2177605f704.slice/crio-f605902602a8dfb9a6c920440583f3e453906d8a8bfe5ab5aa0d8c252c0ab0a3 WatchSource:0}: Error finding container f605902602a8dfb9a6c920440583f3e453906d8a8bfe5ab5aa0d8c252c0ab0a3: Status 404 returned error can't find the container with id f605902602a8dfb9a6c920440583f3e453906d8a8bfe5ab5aa0d8c252c0ab0a3
Nov 03 17:33:02.082626 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:02.082602    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a0388c_1177_45a8_8eb8_92ef23d8f0bf.slice/crio-4d269573568f76713c6eefcebcd2d01ee83ec9228ce3677b68c83ea0a3521991 WatchSource:0}: Error finding container 4d269573568f76713c6eefcebcd2d01ee83ec9228ce3677b68c83ea0a3521991: Status 404 returned error can't find the container with id 4d269573568f76713c6eefcebcd2d01ee83ec9228ce3677b68c83ea0a3521991
Nov 03 17:33:02.436823 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.436786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:02.437025 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.436835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:02.437025 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.436868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:02.437025 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.436982    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:02.437025 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.436989    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:02.437025 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.436997    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:02.437025 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.437021    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:02.437319 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.437035    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v5czc for pod openshift-network-diagnostics/network-check-target-7v8km: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:02.437319 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.437043    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret podName:9b82e04a-d3d2-43de-ad90-abccdeb48e7e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:03.437026933 +0000 UTC m=+4.209482685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret") pod "global-pull-secret-syncer-648bg" (UID: "9b82e04a-d3d2-43de-ad90-abccdeb48e7e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:02.437319 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.437057    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:03.43705108 +0000 UTC m=+4.209506812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:02.437319 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:02.437080    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc podName:44f3aa72-92ab-4dff-9c6a-2c78626e8b50 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:03.437061962 +0000 UTC m=+4.209517718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5czc" (UniqueName: "kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc") pod "network-check-target-7v8km" (UID: "44f3aa72-92ab-4dff-9c6a-2c78626e8b50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:02.712940 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.712837    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl" event={"ID":"1e373a48-c67d-4cd3-8837-166a41802085","Type":"ContainerStarted","Data":"42e50590bf2ca76e52dcd453d3f6036937af7061297cd967de8296b60d4c772c"}
Nov 03 17:33:02.714514 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.714486    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"473ffa27f4d4206455c8e944d4ee676dcf7c1786073575e82db66ead79e62989"}
Nov 03 17:33:02.715858 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.715830    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qrt7" event={"ID":"01b8876a-2097-40ee-9b09-4ec46606a7b1","Type":"ContainerStarted","Data":"f06fe48aa384d0f6d9876177748c36405e6e4845ae0ae70efde4e2c92053636f"}
Nov 03 17:33:02.717404 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.717381    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t4sns" event={"ID":"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf","Type":"ContainerStarted","Data":"4d269573568f76713c6eefcebcd2d01ee83ec9228ce3677b68c83ea0a3521991"}
Nov 03 17:33:02.718642 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.718615    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9n9mf" event={"ID":"d3138bcb-bcf2-49bc-959d-c2177605f704","Type":"ContainerStarted","Data":"f605902602a8dfb9a6c920440583f3e453906d8a8bfe5ab5aa0d8c252c0ab0a3"}
Nov 03 17:33:02.719795 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.719773    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hcf4k" event={"ID":"2c073539-dddd-4fd2-88ee-52fc61b9f50d","Type":"ContainerStarted","Data":"4bf56a416a86b91ee9ee9c5473367cf25ea649e42797f5486653dbef49f71b82"}
Nov 03 17:33:02.720974 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.720949    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vcxvs" event={"ID":"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09","Type":"ContainerStarted","Data":"2ada58c712a4d8da4b6f4215f66659e428cf0b9bd8e38cfbe42c8588f2a307eb"}
Nov 03 17:33:02.722179 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.722158    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-88fkm" event={"ID":"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40","Type":"ContainerStarted","Data":"bd82c239c67a9360494a131b1ec0674a040a585daf4235ebd55469843584b23e"}
Nov 03 17:33:02.723389 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.723368    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerStarted","Data":"f396f99503ac8aef8cf922b88db3e418127442b9ccd4a9ec0d5d0e67ad871675"}
Nov 03 17:33:02.814976 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.814911    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:01 +0000 UTC" deadline="2028-06-26 13:50:35.178788868 +0000 UTC"
Nov 03 17:33:02.814976 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:02.814967    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14420h17m32.363826235s"
Nov 03 17:33:03.444824 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.444786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:03.445057 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.444840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:03.445057 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.444908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:03.445057 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.444959    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:03.445057 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.444982    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:03.445057 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.444991    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v5czc for pod openshift-network-diagnostics/network-check-target-7v8km: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:03.445057 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.445019    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:03.445057 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.445035    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc podName:44f3aa72-92ab-4dff-9c6a-2c78626e8b50 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:05.445023465 +0000 UTC m=+6.217479198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5czc" (UniqueName: "kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc") pod "network-check-target-7v8km" (UID: "44f3aa72-92ab-4dff-9c6a-2c78626e8b50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:03.445514 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.445070    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:05.445054798 +0000 UTC m=+6.217510532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:03.445514 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.445083    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:03.445514 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.445104    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret podName:9b82e04a-d3d2-43de-ad90-abccdeb48e7e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:05.445097752 +0000 UTC m=+6.217553485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret") pod "global-pull-secret-syncer-648bg" (UID: "9b82e04a-d3d2-43de-ad90-abccdeb48e7e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:03.707141 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.706418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:03.707141 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.706541    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:03.707141 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.706628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:03.707141 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.706712    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:03.707141 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.706769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:03.707141 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:03.706837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:03.816299 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.816220    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:01 +0000 UTC" deadline="2028-08-06 20:01:37.264648942 +0000 UTC"
Nov 03 17:33:03.816761 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:03.816308    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15410h28m33.448357784s"
Nov 03 17:33:04.746167 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:04.744825    2576 generic.go:356] "Generic (PLEG): container finished" podID="dbe46139cd173525f608561351491180" containerID="621c5a17322dea1cd41e9e06c2d294287980497158ac977991d28d572613e7b4" exitCode=0
Nov 03 17:33:04.746167 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:04.746048    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"dbe46139cd173525f608561351491180","Type":"ContainerDied","Data":"621c5a17322dea1cd41e9e06c2d294287980497158ac977991d28d572613e7b4"}
Nov 03 17:33:05.470236 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.470203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.470251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.470270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470366    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470410    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret podName:9b82e04a-d3d2-43de-ad90-abccdeb48e7e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:09.470398724 +0000 UTC m=+10.242854458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret") pod "global-pull-secret-syncer-648bg" (UID: "9b82e04a-d3d2-43de-ad90-abccdeb48e7e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470471    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470502    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:09.470490889 +0000 UTC m=+10.242946626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470581    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470596    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470608    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v5czc for pod openshift-network-diagnostics/network-check-target-7v8km: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:05.470651 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.470641    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc podName:44f3aa72-92ab-4dff-9c6a-2c78626e8b50 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:09.470630242 +0000 UTC m=+10.243085982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5czc" (UniqueName: "kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc") pod "network-check-target-7v8km" (UID: "44f3aa72-92ab-4dff-9c6a-2c78626e8b50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:05.707235 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.707205    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:05.707399 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.707328    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:05.707462 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.707399    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:05.707515 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.707473    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:05.707578 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.707537    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:05.707649 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:05.707619    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:05.754709 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.754666    2576 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="d4a5a7de8a4b1700961d06f59bf7caf5aabe2f005eaf9ce1217067bc7e01ba50" exitCode=0
Nov 03 17:33:05.754857 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.754756    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerDied","Data":"d4a5a7de8a4b1700961d06f59bf7caf5aabe2f005eaf9ce1217067bc7e01ba50"}
Nov 03 17:33:05.760515 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:05.760489    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"dbe46139cd173525f608561351491180","Type":"ContainerStarted","Data":"f48ee5c96cdb0e87d3cb5769dd186b2dda1ad614ce93b6ade3691ddba20c0ffb"}
Nov 03 17:33:06.775661 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:06.775627    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"3eefabd2e362fe1b3609803d32f993c7","Type":"ContainerStarted","Data":"cab66d4025d5649fda1a592e56e38df069679cbd8e2f4725af23f061deec71a6"}
Nov 03 17:33:06.895259 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:06.894446    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal" podStartSLOduration=5.894428988 podStartE2EDuration="5.894428988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:06.008425406 +0000 UTC m=+6.780881165" watchObservedRunningTime="2026-11-03 17:33:06.894428988 +0000 UTC m=+7.666884745"
Nov 03 17:33:07.704754 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.704513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:07.704754 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.704565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:07.704754 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:07.704669    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:07.704754 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:07.704756    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:07.705192 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.704810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:07.705192 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:07.704893    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:07.784404 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.783637    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9n9mf" event={"ID":"d3138bcb-bcf2-49bc-959d-c2177605f704","Type":"ContainerStarted","Data":"c957e8574a9ac2376df0ef8aba5688cac24a7a1d46674ff18c8e44619dd25f88"}
Nov 03 17:33:07.791829 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.791754    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t4sns" event={"ID":"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf","Type":"ContainerStarted","Data":"7c3d8f584a06745c5894a4b27db8470fff702121f5c0af9d9d8262bafc7b5562"}
Nov 03 17:33:07.818204 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.818163    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:07.818956 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.818938    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:07.902503 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.902449    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal" podStartSLOduration=6.902428267 podStartE2EDuration="6.902428267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:06.895306828 +0000 UTC m=+7.667762586" watchObservedRunningTime="2026-11-03 17:33:07.902428267 +0000 UTC m=+8.674884022"
Nov 03 17:33:07.902696 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:07.902591    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9n9mf" podStartSLOduration=2.58361465 podStartE2EDuration="7.902582821s" totalImagesPullingTime="5.318968171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:07.902524854 +0000 UTC m=+8.674980612" watchObservedRunningTime="2026-11-03 17:33:07.902582821 +0000 UTC m=+8.675038576"
Nov 03 17:33:08.797918 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:08.797636    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl" event={"ID":"1e373a48-c67d-4cd3-8837-166a41802085","Type":"ContainerStarted","Data":"5ff65e3630e27a81ce22504be2d90ae41d41393000f690ae9169c71ff78aef21"}
Nov 03 17:33:09.505151 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:09.505107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:09.505325 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:09.505212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:09.505325 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:09.505253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:09.505431 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.505398    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:09.506030 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.505469    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret podName:9b82e04a-d3d2-43de-ad90-abccdeb48e7e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:17.505451174 +0000 UTC m=+18.277906909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret") pod "global-pull-secret-syncer-648bg" (UID: "9b82e04a-d3d2-43de-ad90-abccdeb48e7e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:09.506030 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.505925    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:09.506030 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.505984    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:17.505969833 +0000 UTC m=+18.278425567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:09.506030 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.505987    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:09.506030 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.506004    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:09.506030 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.506018    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v5czc for pod openshift-network-diagnostics/network-check-target-7v8km: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:09.506341 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.506063    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc podName:44f3aa72-92ab-4dff-9c6a-2c78626e8b50 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:17.506051525 +0000 UTC m=+18.278507258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5czc" (UniqueName: "kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc") pod "network-check-target-7v8km" (UID: "44f3aa72-92ab-4dff-9c6a-2c78626e8b50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:09.706305 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:09.706275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:09.706479 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.706395    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:09.707046 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:09.707025    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:09.707161 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.707120    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:09.707457 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:09.707307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:09.707457 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:09.707413    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:09.799379 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:09.799307    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:10.692375 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.692349    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:33:10.698436 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.698321    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:33:10.692370849Z","UUID":"f40de5f8-aa89-4536-9deb-7fa9c05aaac2","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:33:10.700855 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.700813    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:33:10.700855 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.700838    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:33:10.803933 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.803669    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hcf4k" event={"ID":"2c073539-dddd-4fd2-88ee-52fc61b9f50d","Type":"ContainerStarted","Data":"75632e92e7a6d13893fa9936229ad0f7fe9a33cd89783f5b2298bd1efb45a4e3"}
Nov 03 17:33:10.806511 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.806483    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl" event={"ID":"1e373a48-c67d-4cd3-8837-166a41802085","Type":"ContainerStarted","Data":"36486e51b6a6138b2cf0213bbfe018b899ae272f3b69010bacda8bb1f90c3fae"}
Nov 03 17:33:10.808289 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.808257    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qrt7" event={"ID":"01b8876a-2097-40ee-9b09-4ec46606a7b1","Type":"ContainerStarted","Data":"f0e9198daa63e08e835b24ed7533418b0a8f3ba20f23de47593a9b652b3c9716"}
Nov 03 17:33:10.923663 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.923468    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t4sns" podStartSLOduration=6.302283258 podStartE2EDuration="10.923454168s" totalImagesPullingTime="4.62117091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:08.022154972 +0000 UTC m=+8.794610726" watchObservedRunningTime="2026-11-03 17:33:10.923454168 +0000 UTC m=+11.695909922"
Nov 03 17:33:10.924128 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:10.924089    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hcf4k" podStartSLOduration=2.541575985 podStartE2EDuration="10.924075928s" totalImagesPullingTime="8.382499943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:10.923381887 +0000 UTC m=+11.695837643" watchObservedRunningTime="2026-11-03 17:33:10.924075928 +0000 UTC m=+11.696531687"
Nov 03 17:33:11.045551 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.045459    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7qrt7" podStartSLOduration=2.936981077 podStartE2EDuration="11.04544364s" totalImagesPullingTime="8.108462563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:11.044780207 +0000 UTC m=+11.817235962" watchObservedRunningTime="2026-11-03 17:33:11.04544364 +0000 UTC m=+11.817899395"
Nov 03 17:33:11.132615 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.132582    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:11.132773 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.132714    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:11.133462 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.133440    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:11.705120 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.705038    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:11.705120 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.705054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:11.705120 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.705087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:11.705401 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:11.705197    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:11.705401 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:11.705335    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:11.705495 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:11.705424    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:11.813503 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.813463    2576 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="41aaf4fbd1bb0f5081f95044019f870837a49ee3557eb4176a0f66bb7f5f7590" exitCode=0
Nov 03 17:33:11.814027 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:11.813643    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerDied","Data":"41aaf4fbd1bb0f5081f95044019f870837a49ee3557eb4176a0f66bb7f5f7590"}
Nov 03 17:33:12.818409 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:12.818367    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl" event={"ID":"1e373a48-c67d-4cd3-8837-166a41802085","Type":"ContainerStarted","Data":"fac35a885d688bc16ef5f82b5c9b744c27f119dec8514f41e384246fd23d51ae"}
Nov 03 17:33:12.820060 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:12.820033    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-88fkm" event={"ID":"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40","Type":"ContainerStarted","Data":"3706f1563e734019a06580a05b9dac2416e53e38217e38abeeb4125454016965"}
Nov 03 17:33:13.067841 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.067790    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl" podStartSLOduration=3.417533202 podStartE2EDuration="13.067770373s" totalImagesPullingTime="9.650237171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:12.942424044 +0000 UTC m=+13.714879801" watchObservedRunningTime="2026-11-03 17:33:13.067770373 +0000 UTC m=+13.840226132"
Nov 03 17:33:13.068088 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.068050    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-88fkm" podStartSLOduration=4.572209509 podStartE2EDuration="13.068036708s" totalImagesPullingTime="8.495827199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:13.067317133 +0000 UTC m=+13.839772887" watchObservedRunningTime="2026-11-03 17:33:13.068036708 +0000 UTC m=+13.840492463"
Nov 03 17:33:13.704333 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.704301    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:13.704487 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:13.704412    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:13.704712 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.704698    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:13.704781 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:13.704767    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:13.704968 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.704943    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:13.705087 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:13.705060    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:13.824110 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.824051    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vcxvs" event={"ID":"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09","Type":"ContainerStarted","Data":"0d637540299ff68b89c35727a18b997e835674c9d5555b910569717a35ee2ae8"}
Nov 03 17:33:13.827479 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.826625    2576 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="890fd62c93f37e68acd7b933ccd3dba0bc23600519387acfe980b11c1b819ed6" exitCode=0
Nov 03 17:33:13.827479 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.826651    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerDied","Data":"890fd62c93f37e68acd7b933ccd3dba0bc23600519387acfe980b11c1b819ed6"}
Nov 03 17:33:13.947577 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:13.947486    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vcxvs" podStartSLOduration=3.166025101 podStartE2EDuration="13.94746926s" totalImagesPullingTime="10.781444159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:13.946508619 +0000 UTC m=+14.718964372" watchObservedRunningTime="2026-11-03 17:33:13.94746926 +0000 UTC m=+14.719925020"
Nov 03 17:33:15.704590 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:15.704336    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:15.705041 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:15.704427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:15.705041 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:15.704630    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:15.705041 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:15.704742    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:15.705041 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:15.704487    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:15.705041 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:15.704862    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:15.833435 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:15.833406    2576 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="62d6687be3567cbd932cfaa54b357987c9d141e748a2ef352794284c8766b204" exitCode=0
Nov 03 17:33:15.833604 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:15.833449    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerDied","Data":"62d6687be3567cbd932cfaa54b357987c9d141e748a2ef352794284c8766b204"}
Nov 03 17:33:17.572103 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.571848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572007    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.572168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572208    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret podName:9b82e04a-d3d2-43de-ad90-abccdeb48e7e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:33.572185213 +0000 UTC m=+34.344640960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret") pod "global-pull-secret-syncer-648bg" (UID: "9b82e04a-d3d2-43de-ad90-abccdeb48e7e") : object "kube-system"/"original-pull-secret" not registered
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572251    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.572260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572305    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:33.572289475 +0000 UTC m=+34.344745222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572381    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572397    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572410    2576 projected.go:196] Error preparing data for projected volume kube-api-access-v5czc for pod openshift-network-diagnostics/network-check-target-7v8km: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:17.572581 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.572457    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc podName:44f3aa72-92ab-4dff-9c6a-2c78626e8b50 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:33.57244427 +0000 UTC m=+34.344900003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v5czc" (UniqueName: "kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc") pod "network-check-target-7v8km" (UID: "44f3aa72-92ab-4dff-9c6a-2c78626e8b50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 17:33:17.704901 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.704798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:17.704901 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.704839    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:17.704901 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.704798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:17.705166 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.704945    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:17.705166 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.705035    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:17.705166 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:17.705143    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:17.842203 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.842168    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"964c5d82e899dd268f222c47180a4411726c484f60231431e902048f69c1303d"}
Nov 03 17:33:17.842312 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.842253    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"52923c2cc3a42d4df263ea22b1737db31fc9d6984e5818939bbb74ede9277adf"}
Nov 03 17:33:17.842312 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.842285    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"1862831687fc9260e3b1a1bab6e69b78f8de670762a76b58a1bb525919a1255f"}
Nov 03 17:33:17.842312 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.842311    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"6fefd550d09b8c463934584fdeb4922f4726b14e98ca3f263d5167604e9fd6a6"}
Nov 03 17:33:17.842426 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.842338    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"1535b3d7575c9f25cb2fc53dbcc130dbca608350156eaf7dd2442ba607217408"}
Nov 03 17:33:17.842426 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:17.842364    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"62c6a7482fdf6182f15473334bac9ae340e6c71e87ec5effa8f70e0399137564"}
Nov 03 17:33:18.846575 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:18.846544    2576 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="da659913732b37413354e027110ab62c2ff043bb5fd463b5645a134fa18c41c7" exitCode=0
Nov 03 17:33:18.847104 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:18.846598    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerDied","Data":"da659913732b37413354e027110ab62c2ff043bb5fd463b5645a134fa18c41c7"}
Nov 03 17:33:19.707636 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:19.706606    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:19.707636 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:19.706772    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:19.707636 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:19.706820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:19.707636 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:19.707516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:19.707946 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:19.707645    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:19.707946 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:19.707735    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:19.854092 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:19.854000    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"df85af29b5cc6c3ee310d2593f37ac9e0c55d20235282641eeb76ec3ecb2cb2f"}
Nov 03 17:33:19.856982 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:19.856959    2576 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="f5a11f2da2695f93a3ffc769c566ab98b505e0310afcffefe28e016fd56e8bc5" exitCode=0
Nov 03 17:33:19.857114 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:19.856986    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerDied","Data":"f5a11f2da2695f93a3ffc769c566ab98b505e0310afcffefe28e016fd56e8bc5"}
Nov 03 17:33:20.861863 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:20.861821    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" event={"ID":"8222297a-ec44-495a-b6bd-6b58501aadd4","Type":"ContainerStarted","Data":"dff4d68f5f192fea883647ed712ff0a3eb940cc4b13cd33f8006768cff238232"}
Nov 03 17:33:20.989017 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:20.988967    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vfd7n" podStartSLOduration=7.884601665 podStartE2EDuration="20.988952886s" totalImagesPullingTime="13.104351221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:20.988472927 +0000 UTC m=+21.760928682" watchObservedRunningTime="2026-11-03 17:33:20.988952886 +0000 UTC m=+21.761408642"
Nov 03 17:33:21.704250 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:21.704216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:21.704401 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:21.704276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:21.704401 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:21.704293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:21.704401 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:21.704366    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:21.704598 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:21.704466    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:21.704598 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:21.704535    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:21.867940 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:21.867613    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" event={"ID":"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0","Type":"ContainerStarted","Data":"e45ddb3cc39074601b98524ef572d0f2e36f8457daece95eea098e7609db9793"}
Nov 03 17:33:22.102333 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:22.102270    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf" podStartSLOduration=7.064707992 podStartE2EDuration="22.102256539s" totalImagesPullingTime="15.037548547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:22.101768113 +0000 UTC m=+22.874223881" watchObservedRunningTime="2026-11-03 17:33:22.102256539 +0000 UTC m=+22.874712293"
Nov 03 17:33:22.870033 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:22.870008    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:22.870472 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:22.870362    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:22.870472 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:22.870438    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:22.887890 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:22.887847    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:22.888081 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:22.888065    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:23.707766 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:23.707735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:23.707939 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:23.707811    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:23.707939 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:23.707835    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:23.708009 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:23.707941    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:23.708009 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:23.707968    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:23.708071 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:23.708028    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:23.872670 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:23.872642    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:24.463707 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:24.463640    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-648bg"]
Nov 03 17:33:24.463846 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:24.463741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:24.463846 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:24.463831    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:24.465407 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:24.465382    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7v8km"]
Nov 03 17:33:24.465547 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:24.465472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:24.465613 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:24.465561    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:24.466486 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:24.466463    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ccdq"]
Nov 03 17:33:24.466590 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:24.466543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:24.466628 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:24.466613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:24.874922 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:24.874836    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:25.704657 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:25.704624    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:25.704780 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:25.704671    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:25.704780 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:25.704763    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:25.704903 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:25.704861    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:25.739808 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:25.739784    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:26.704128 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:26.704100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:26.704463 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:26.704204    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:27.704424 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:27.704396    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:27.704424 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:27.704416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:27.704984 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:27.704492    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:27.704984 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:27.704652    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:28.704062 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:28.704033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:28.704241 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:28.704144    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2ccdq" podUID="c85b087b-5e24-45d0-abe3-d8e72dbbb74e"
Nov 03 17:33:29.704733 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:29.704701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:29.705201 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:29.704859    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7v8km" podUID="44f3aa72-92ab-4dff-9c6a-2c78626e8b50"
Nov 03 17:33:29.705201 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:29.704898    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:29.705201 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:29.705005    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-648bg" podUID="9b82e04a-d3d2-43de-ad90-abccdeb48e7e"
Nov 03 17:33:29.992507 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:29.992285    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeReady"
Nov 03 17:33:29.992640 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:29.992563    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 17:33:30.113814 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.112514    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6shfz"]
Nov 03 17:33:30.116641 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.116615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.161268 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.161237    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4w9qr"]
Nov 03 17:33:30.164697 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.164677    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6shfz"]
Nov 03 17:33:30.164827 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.164769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.171463 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.171443    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:33:30.171688 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.171672    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:33:30.176312 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.176296    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:33:30.176449 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.176436    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jj5j\""
Nov 03 17:33:30.228075 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.228047    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4w9qr"]
Nov 03 17:33:30.266856 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.266788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.266856 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.266830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f2ccd-9e73-4f22-8370-409ca511aeb3-tmp-dir\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.267030 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.266951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstsl\" (UniqueName: \"kubernetes.io/projected/c42f2ccd-9e73-4f22-8370-409ca511aeb3-kube-api-access-pstsl\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.267030 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.266997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42f2ccd-9e73-4f22-8370-409ca511aeb3-config-volume\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.267030 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.267022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1758fefb-559e-433d-a077-bb4b574853e1-cert\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.267312 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.267073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnnd\" (UniqueName: \"kubernetes.io/projected/1758fefb-559e-433d-a077-bb4b574853e1-kube-api-access-lbnnd\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.271738 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.271719    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:33:30.276289 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.276273    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:33:30.276378 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.276329    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2pwfx\""
Nov 03 17:33:30.368247 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.368247 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f2ccd-9e73-4f22-8370-409ca511aeb3-tmp-dir\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.368433 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstsl\" (UniqueName: \"kubernetes.io/projected/c42f2ccd-9e73-4f22-8370-409ca511aeb3-kube-api-access-pstsl\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.368433 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42f2ccd-9e73-4f22-8370-409ca511aeb3-config-volume\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.368433 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1758fefb-559e-433d-a077-bb4b574853e1-cert\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.368433 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnnd\" (UniqueName: \"kubernetes.io/projected/1758fefb-559e-433d-a077-bb4b574853e1-kube-api-access-lbnnd\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.368433 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:30.368382    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:30.368696 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:30.368462    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.868439499 +0000 UTC m=+31.640895249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:30.368696 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368608    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f2ccd-9e73-4f22-8370-409ca511aeb3-tmp-dir\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.368885 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.368848    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42f2ccd-9e73-4f22-8370-409ca511aeb3-config-volume\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.372527 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.372508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1758fefb-559e-433d-a077-bb4b574853e1-cert\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.431828 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.431795    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstsl\" (UniqueName: \"kubernetes.io/projected/c42f2ccd-9e73-4f22-8370-409ca511aeb3-kube-api-access-pstsl\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.431986 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.431905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnnd\" (UniqueName: \"kubernetes.io/projected/1758fefb-559e-433d-a077-bb4b574853e1-kube-api-access-lbnnd\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.704517 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.704431    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:30.728483 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.728450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:30.760081 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.760051    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zl7j\""
Nov 03 17:33:30.760081 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.760051    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:33:30.872350 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:30.872321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:30.872510 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:30.872443    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:30.872510 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:30.872510    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:31.872494403 +0000 UTC m=+32.644950136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:31.171605 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.171515    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6shfz"]
Nov 03 17:33:31.175636 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:31.175597    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1758fefb_559e_433d_a077_bb4b574853e1.slice/crio-70ebd968fb3a7421674a5187a2d775e941937203b782036231e36027ef62143c WatchSource:0}: Error finding container 70ebd968fb3a7421674a5187a2d775e941937203b782036231e36027ef62143c: Status 404 returned error can't find the container with id 70ebd968fb3a7421674a5187a2d775e941937203b782036231e36027ef62143c
Nov 03 17:33:31.704238 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.704206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:31.704419 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.704216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:31.759612 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.759575    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:33:31.759612 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.759586    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:33:31.760049 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.759712    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-glkrd\""
Nov 03 17:33:31.760049 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.759788    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:33:31.760049 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.759790    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xhql4\""
Nov 03 17:33:31.877525 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.877495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:31.877645 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:31.877600    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:31.877697 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:31.877652    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:33.877638301 +0000 UTC m=+34.650094036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:31.891474 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:31.891444    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6shfz" event={"ID":"1758fefb-559e-433d-a077-bb4b574853e1","Type":"ContainerStarted","Data":"70ebd968fb3a7421674a5187a2d775e941937203b782036231e36027ef62143c"}
Nov 03 17:33:33.588319 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.588285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:33.588934 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.588352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:33.588934 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.588389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:33.588934 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:33.588456    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 17:33:33.588934 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:33.588542    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:34:05.588520267 +0000 UTC m=+66.360976021 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : secret "metrics-daemon-secret" not found
Nov 03 17:33:33.591455 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.591425    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:33.593527 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.593479    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:33.842323 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.842242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:33.858440 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.858412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:33.890456 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:33.890420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:33.890589 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:33.890575    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:33.890649 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:33.890631    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:37.890613181 +0000 UTC m=+38.663068923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:34.244620 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:34.244584    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7v8km"]
Nov 03 17:33:34.248937 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:34.248899    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f3aa72_92ab_4dff_9c6a_2c78626e8b50.slice/crio-3445fdb14af949fee6f44944ec6c2423afc01bf99cafb7253e5737ad58d0b29a WatchSource:0}: Error finding container 3445fdb14af949fee6f44944ec6c2423afc01bf99cafb7253e5737ad58d0b29a: Status 404 returned error can't find the container with id 3445fdb14af949fee6f44944ec6c2423afc01bf99cafb7253e5737ad58d0b29a
Nov 03 17:33:34.292663 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:34.292632    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-648bg"]
Nov 03 17:33:34.295979 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:34.295948    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b82e04a_d3d2_43de_ad90_abccdeb48e7e.slice/crio-36ecfe7b235090f1a3154276fb9d45099a4d2a1b3a4b78ab6283fd20c550ef2b WatchSource:0}: Error finding container 36ecfe7b235090f1a3154276fb9d45099a4d2a1b3a4b78ab6283fd20c550ef2b: Status 404 returned error can't find the container with id 36ecfe7b235090f1a3154276fb9d45099a4d2a1b3a4b78ab6283fd20c550ef2b
Nov 03 17:33:34.899615 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:34.899578    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7v8km" event={"ID":"44f3aa72-92ab-4dff-9c6a-2c78626e8b50","Type":"ContainerStarted","Data":"3445fdb14af949fee6f44944ec6c2423afc01bf99cafb7253e5737ad58d0b29a"}
Nov 03 17:33:34.900727 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:34.900706    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-648bg" event={"ID":"9b82e04a-d3d2-43de-ad90-abccdeb48e7e","Type":"ContainerStarted","Data":"36ecfe7b235090f1a3154276fb9d45099a4d2a1b3a4b78ab6283fd20c550ef2b"}
Nov 03 17:33:35.905532 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:35.905315    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6shfz" event={"ID":"1758fefb-559e-433d-a077-bb4b574853e1","Type":"ContainerStarted","Data":"b46de2849a4aa421399e01afce4c70d145cfe6418655eaa33b7ac33b032157e0"}
Nov 03 17:33:36.026254 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.026195    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6shfz" podStartSLOduration=2.214289802 podStartE2EDuration="6.026174216s" totalImagesPullingTime="3.811884414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:36.025265969 +0000 UTC m=+36.797721725" watchObservedRunningTime="2026-11-03 17:33:36.026174216 +0000 UTC m=+36.798629974"
Nov 03 17:33:36.287753 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.287713    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-82x2r"]
Nov 03 17:33:36.309683 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.309657    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.364735 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.364701    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:33:36.364735 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.364711    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:33:36.364947 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.364751    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:33:36.365058 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.365040    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:33:36.365132 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.365114    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bstfb\""
Nov 03 17:33:36.383401 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.383373    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-82x2r"]
Nov 03 17:33:36.409924 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.409805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458s7\" (UniqueName: \"kubernetes.io/projected/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-api-access-458s7\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.409924 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.409855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-crio-socket\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.410127 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.409967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-data-volume\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.410127 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.410088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.410239 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.410159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.510463 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.510424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.510649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.510497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.510649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.510525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458s7\" (UniqueName: \"kubernetes.io/projected/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-api-access-458s7\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.510649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.510553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-crio-socket\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.510649 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:36.510567    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:33:36.510649 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.510588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-data-volume\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.510649 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:36.510651    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:37.010632114 +0000 UTC m=+37.783087861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : configmap "kube-rbac-proxy" not found
Nov 03 17:33:36.511029 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.510971    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-data-volume\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.511124 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:36.511102    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:36.511247 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:36.511156    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:37.011141094 +0000 UTC m=+37.783596830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:36.511247 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.511186    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-crio-socket\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.541571 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.541496    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fssvd-2gvm4"]
Nov 03 17:33:36.558925 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.558897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.575865 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.575807    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458s7\" (UniqueName: \"kubernetes.io/projected/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-api-access-458s7\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:36.604579 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.604542    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fssvd-2gvm4"]
Nov 03 17:33:36.614165 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.614092    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 17:33:36.665029 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.665000    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-htgnv\""
Nov 03 17:33:36.665476 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.665452    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 17:33:36.712785 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.712752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.712985 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.712817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5l8w\" (UniqueName: \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.712985 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.712944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.713104 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.713007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.813486 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.813392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.813486 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.813480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.813698 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.813516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.813698 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.813558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5l8w\" (UniqueName: \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.814260 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.814227    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.815005 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.814980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.815190 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.815172    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.877860 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.877828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5l8w\" (UniqueName: \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:36.886489 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:36.886466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:37.016141 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.015321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:37.016141 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.015448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:37.016141 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:37.015547    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:33:37.016141 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:37.015606    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:38.015587874 +0000 UTC m=+38.788043610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : configmap "kube-rbac-proxy" not found
Nov 03 17:33:37.016141 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:37.015852    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:37.016141 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:37.015913    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:38.015903146 +0000 UTC m=+38.788358878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:37.300125 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.300087    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fssvd-2gvm4"]
Nov 03 17:33:37.303729 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:37.303698    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bef3d2_aed3_4a2a_bf78_6f6a2febe2f9.slice/crio-62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab WatchSource:0}: Error finding container 62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab: Status 404 returned error can't find the container with id 62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab
Nov 03 17:33:37.370993 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.370964    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"]
Nov 03 17:33:37.396928 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.396571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"
Nov 03 17:33:37.442193 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.442161    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"]
Nov 03 17:33:37.453521 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.453494    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:37.453653 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.453600    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 17:33:37.453790 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.453776    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-df4kg\""
Nov 03 17:33:37.519562 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.519529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56t79\" (UniqueName: \"kubernetes.io/projected/60cede5f-b376-45dd-b774-feb3598cf479-kube-api-access-56t79\") pod \"migrator-66df5d496b-xjvqh\" (UID: \"60cede5f-b376-45dd-b774-feb3598cf479\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"
Nov 03 17:33:37.620727 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.620362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56t79\" (UniqueName: \"kubernetes.io/projected/60cede5f-b376-45dd-b774-feb3598cf479-kube-api-access-56t79\") pod \"migrator-66df5d496b-xjvqh\" (UID: \"60cede5f-b376-45dd-b774-feb3598cf479\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"
Nov 03 17:33:37.686729 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.686700    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56t79\" (UniqueName: \"kubernetes.io/projected/60cede5f-b376-45dd-b774-feb3598cf479-kube-api-access-56t79\") pod \"migrator-66df5d496b-xjvqh\" (UID: \"60cede5f-b376-45dd-b774-feb3598cf479\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"
Nov 03 17:33:37.724995 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.724958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"
Nov 03 17:33:37.911433 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.911362    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fssvd-2gvm4" event={"ID":"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9","Type":"ContainerStarted","Data":"62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab"}
Nov 03 17:33:37.922690 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:37.922661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:37.922822 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:37.922802    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:37.922891 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:37.922853    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.922839962 +0000 UTC m=+46.695295696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:38.023754 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.023714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:38.024207 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.023812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:38.024207 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:38.023892    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:38.024207 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:38.023918    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:33:38.024207 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:38.023973    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:40.023952818 +0000 UTC m=+40.796408552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:38.024207 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:38.023995    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:40.023984317 +0000 UTC m=+40.796440049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : configmap "kube-rbac-proxy" not found
Nov 03 17:33:38.115663 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.115633    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"]
Nov 03 17:33:38.119022 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:38.118994    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cede5f_b376_45dd_b774_feb3598cf479.slice/crio-8c685e7e7c4ae9e5c08c3a4572ccead9e81a17d3f49144c7d75906a61c876ce2 WatchSource:0}: Error finding container 8c685e7e7c4ae9e5c08c3a4572ccead9e81a17d3f49144c7d75906a61c876ce2: Status 404 returned error can't find the container with id 8c685e7e7c4ae9e5c08c3a4572ccead9e81a17d3f49144c7d75906a61c876ce2
Nov 03 17:33:38.627654 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.627566    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-gjgn4"]
Nov 03 17:33:38.649420 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.649394    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.691644 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.691613    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-gjgn4"]
Nov 03 17:33:38.704658 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.704632    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 17:33:38.704802 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.704684    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 17:33:38.704802 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.704708    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 17:33:38.704952 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.704937    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 17:33:38.705072 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.705052    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tz4q7\""
Nov 03 17:33:38.730381 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.730350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9699d94f-597e-4612-a469-3e657bfc9a55-signing-key\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.730511 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.730397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm67z\" (UniqueName: \"kubernetes.io/projected/9699d94f-597e-4612-a469-3e657bfc9a55-kube-api-access-nm67z\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.730511 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.730432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9699d94f-597e-4612-a469-3e657bfc9a55-signing-cabundle\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.831805 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.831773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9699d94f-597e-4612-a469-3e657bfc9a55-signing-key\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.832063 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.831819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm67z\" (UniqueName: \"kubernetes.io/projected/9699d94f-597e-4612-a469-3e657bfc9a55-kube-api-access-nm67z\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.832063 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.831852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9699d94f-597e-4612-a469-3e657bfc9a55-signing-cabundle\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.832656 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.832629    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9699d94f-597e-4612-a469-3e657bfc9a55-signing-cabundle\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.834931 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.834912    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9699d94f-597e-4612-a469-3e657bfc9a55-signing-key\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.896985 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.896925    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm67z\" (UniqueName: \"kubernetes.io/projected/9699d94f-597e-4612-a469-3e657bfc9a55-kube-api-access-nm67z\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:38.915525 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.915491    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh" event={"ID":"60cede5f-b376-45dd-b774-feb3598cf479","Type":"ContainerStarted","Data":"8c685e7e7c4ae9e5c08c3a4572ccead9e81a17d3f49144c7d75906a61c876ce2"}
Nov 03 17:33:38.918591 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.918560    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7v8km" event={"ID":"44f3aa72-92ab-4dff-9c6a-2c78626e8b50","Type":"ContainerStarted","Data":"8ab7fb0c863529dbb61f67f2c186ae0f629f4c280fe7d9d7977d054c2bef6286"}
Nov 03 17:33:38.918748 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.918733    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:38.973705 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:38.973673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:39.040477 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:39.039244    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7v8km" podStartSLOduration=34.976865154 podStartE2EDuration="39.039226587s" totalImagesPullingTime="4.062361433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:39.038923302 +0000 UTC m=+39.811379050" watchObservedRunningTime="2026-11-03 17:33:39.039226587 +0000 UTC m=+39.811682338"
Nov 03 17:33:39.393683 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:39.393651    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-gjgn4"]
Nov 03 17:33:39.398508 ip-10-0-131-172 kubenswrapper[2576]: W1103 17:33:39.398468    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9699d94f_597e_4612_a469_3e657bfc9a55.slice/crio-6d4dbe0346f4bbd212c9bb75aef2d3a2661ea25537ad136bbc300f14477518fc WatchSource:0}: Error finding container 6d4dbe0346f4bbd212c9bb75aef2d3a2661ea25537ad136bbc300f14477518fc: Status 404 returned error can't find the container with id 6d4dbe0346f4bbd212c9bb75aef2d3a2661ea25537ad136bbc300f14477518fc
Nov 03 17:33:39.923115 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:39.923066    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4" event={"ID":"9699d94f-597e-4612-a469-3e657bfc9a55","Type":"ContainerStarted","Data":"6d4dbe0346f4bbd212c9bb75aef2d3a2661ea25537ad136bbc300f14477518fc"}
Nov 03 17:33:40.049603 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:40.049568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:40.050003 ip-10-0-131-172 kubenswrapper[2576]: I1103 17:33:40.049622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:40.050003 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:40.049708    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:33:40.050003 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:40.049772    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:40.050003 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:40.049781    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.049759155 +0000 UTC m=+44.822214888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : configmap "kube-rbac-proxy" not found
Nov 03 17:33:40.050003 ip-10-0-131-172 kubenswrapper[2576]: E1103 17:33:40.049832    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.049820446 +0000 UTC m=+44.822276179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:40.468431 ip-10-0-131-172 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 17:33:40.481309 ip-10-0-131-172 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 17:33:40.481486 ip-10-0-131-172 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 17:33:40.481534 ip-10-0-131-172 systemd[1]: kubelet.service: Consumed 2.452s CPU time.
Nov 03 17:33:40.496369 ip-10-0-131-172 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 17:33:40.583961 ip-10-0-131-172 kubenswrapper[4493]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:33:40.583961 ip-10-0-131-172 kubenswrapper[4493]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 17:33:40.583961 ip-10-0-131-172 kubenswrapper[4493]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:33:40.583961 ip-10-0-131-172 kubenswrapper[4493]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 17:33:40.586600 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586517    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.586600 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586597    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.586600 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586604    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.586600 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586609    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586613    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586617    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586621    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586625    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586628    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586632    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586635    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586639    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586643    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586647    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586651    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586654    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586662    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586667    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586671    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586675    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586679    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586682    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586686    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.586887 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586690    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586694    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586698    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586702    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586706    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586710    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586714    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586719    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586723    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586726    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586730    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586734    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586739    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586744    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586749    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586753    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586757    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586761    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586766    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.587744 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586770    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586774    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586778    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586782    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586786    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586792    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586796    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586801    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586805    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586809    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586813    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586817    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586822    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586826    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586830    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586835    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586839    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586844    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586849    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586853    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.588592 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586860    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586865    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586889    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586894    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586899    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586903    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586907    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586911    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586915    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586920    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586924    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586928    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586932    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586937    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586941    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586945    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586949    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586955    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586960    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586964    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.589510 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586968    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586973    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586977    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586981    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586989    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.586996    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587001    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587005    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587010    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587016    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587020    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587025    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587029    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587033    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587038    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587042    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587047    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587051    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587055    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.590411 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587059    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587064    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587068    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587072    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.587076    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588395    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588406    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588411    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588416    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588420    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588425    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588429    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588434    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588438    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588443    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588447    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588451    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588458    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588462    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588466    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.591340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588471    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588475    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588479    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588484    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588488    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588492    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588496    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588500    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588505    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588509    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588513    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588519    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588523    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588529    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588535    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588540    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588545    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588549    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588553    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.592277 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588557    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588561    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588566    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588570    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588574    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588579    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588591    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588595    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588600    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588604    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588608    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588613    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588617    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588621    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588624    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588628    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588633    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588637    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588667    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588673    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.593082 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588678    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588682    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588686    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588690    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588694    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588698    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588703    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588707    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588711    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588715    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588719    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588724    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588728    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588732    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588737    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588741    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588746    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588750    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588754    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.593603 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588759    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588763    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588768    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588774    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588780    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588785    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588789    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588794    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588799    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588804    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588808    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588812    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588816    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588820    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588825    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588829    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588833    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588837    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588842    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.594087 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588846    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588850    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588854    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588858    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588863    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588886    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588891    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588895    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588899    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588905    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588909    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588913    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588917    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.588921    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589740    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589749    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589754    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589759    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589763    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589767    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.594548 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589775    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589785    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589790    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589795    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589800    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589805    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589809    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589813    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589817    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589821    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589826    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589830    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589835    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589840    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589844    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589848    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589853    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589857    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589861    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589865    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.595095 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589890    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589894    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589899    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589903    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589907    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589911    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589917    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589921    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589925    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589932    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589937    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589942    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589947    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589953    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589959    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589964    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589968    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589973    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589977    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.595575 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589982    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589986    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589990    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589994    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.589998    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590002    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590007    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590011    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590015    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590019    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590023    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590033    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590037    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590041    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590045    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590049    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590054    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590058    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590064    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.596059 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590068    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590074    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590078    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590082    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590086    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590090    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590095    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590100    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590105    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590109    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590114    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590118    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590122    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590126    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590130    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590135    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590139    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590143    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590147    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.596499 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590151    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590156    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590160    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590164    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590168    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590172    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590177    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590181    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590186    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590190    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590194    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590198    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590203    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590207    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590211    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590215    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590220    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590224    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590228    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590232    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.597022 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590239    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590243    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.590248    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590382    4493 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590403    4493 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590418    4493 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590429    4493 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590436    4493 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590440    4493 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590447    4493 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590454    4493 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590458    4493 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590463    4493 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590469    4493 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590474    4493 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590479    4493 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590483    4493 flags.go:64] FLAG: --cgroup-root=""
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590487    4493 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590492    4493 flags.go:64] FLAG: --client-ca-file=""
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590497    4493 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590502    4493 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590508    4493 flags.go:64] FLAG: --cluster-domain=""
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590513    4493 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 17:33:40.597534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590518    4493 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590523    4493 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590528    4493 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590534    4493 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590539    4493 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590546    4493 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590551    4493 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590556    4493 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590561    4493 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590567    4493 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590572    4493 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590577    4493 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590584    4493 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590589    4493 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590594    4493 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590599    4493 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590604    4493 flags.go:64] FLAG: --enable-server="true"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590608    4493 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590615    4493 flags.go:64] FLAG: --event-burst="100"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590620    4493 flags.go:64] FLAG: --event-qps="50"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590625    4493 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590629    4493 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590633    4493 flags.go:64] FLAG: --eviction-hard=""
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590640    4493 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 17:33:40.598114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590644    4493 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590649    4493 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590654    4493 flags.go:64] FLAG: --eviction-soft=""
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590658    4493 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590663    4493 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590668    4493 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590674    4493 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590683    4493 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590687    4493 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590692    4493 flags.go:64] FLAG: --feature-gates=""
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590698    4493 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590703    4493 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590708    4493 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590712    4493 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590727    4493 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590733    4493 flags.go:64] FLAG: --help="false"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590738    4493 flags.go:64] FLAG: --hostname-override="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590745    4493 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590750    4493 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590754    4493 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590760    4493 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590765    4493 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590770    4493 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 17:33:40.598799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590774    4493 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590779    4493 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590783    4493 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590788    4493 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590793    4493 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590797    4493 flags.go:64] FLAG: --kube-reserved=""
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590802    4493 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590806    4493 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590811    4493 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590816    4493 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590820    4493 flags.go:64] FLAG: --lock-file=""
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590825    4493 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590829    4493 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590834    4493 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590843    4493 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590848    4493 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590855    4493 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590859    4493 flags.go:64] FLAG: --logging-format="text"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590863    4493 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590886    4493 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590891    4493 flags.go:64] FLAG: --manifest-url=""
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590896    4493 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590909    4493 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590914    4493 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590927    4493 flags.go:64] FLAG: --max-pods="110"
Nov 03 17:33:40.599374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590932    4493 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590939    4493 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590943    4493 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590948    4493 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590953    4493 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590957    4493 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590962    4493 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590973    4493 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590978    4493 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590983    4493 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590987    4493 flags.go:64] FLAG: --pod-cidr=""
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590992    4493 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.590996    4493 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591001    4493 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591005    4493 flags.go:64] FLAG: --port="10250"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591010    4493 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591015    4493 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-04294316c535a5d0e"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591020    4493 flags.go:64] FLAG: --qos-reserved=""
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591025    4493 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591029    4493 flags.go:64] FLAG: --register-node="true"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591034    4493 flags.go:64] FLAG: --register-with-taints=""
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591040    4493 flags.go:64] FLAG: --registry-burst="10"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591045    4493 flags.go:64] FLAG: --registry-qps="5"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591052    4493 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591056    4493 flags.go:64] FLAG: --reserved-memory=""
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591063    4493 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 17:33:40.599988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591067    4493 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591072    4493 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591077    4493 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591081    4493 flags.go:64] FLAG: --runonce="false"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591087    4493 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591091    4493 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591104    4493 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591110    4493 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591115    4493 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591120    4493 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591124    4493 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591129    4493 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591134    4493 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591139    4493 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591144    4493 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591148    4493 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591153    4493 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591158    4493 flags.go:64] FLAG: --system-cgroups=""
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591162    4493 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591171    4493 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591175    4493 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591180    4493 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591186    4493 flags.go:64] FLAG: --tls-min-version=""
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591190    4493 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591195    4493 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 17:33:40.600657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591199    4493 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591204    4493 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591208    4493 flags.go:64] FLAG: --v="2"
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591215    4493 flags.go:64] FLAG: --version="false"
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591223    4493 flags.go:64] FLAG: --vmodule=""
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591229    4493 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.591235    4493 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591497    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591506    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591511    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591516    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591521    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591525    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591529    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591537    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591542    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591546    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591551    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591555    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591560    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591564    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.601376 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591568    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591572    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591576    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591581    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591585    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591589    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591593    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591597    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591602    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591606    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591610    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591614    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591618    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591623    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591627    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591635    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591639    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591643    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591647    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591651    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.602293 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591655    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591659    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591663    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591668    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591672    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591678    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591683    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591687    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591691    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591695    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591700    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591704    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591708    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591712    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591716    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591720    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591724    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591728    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591732    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591736    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.603015 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591740    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591745    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591749    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591753    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591757    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591762    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591768    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591775    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591780    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591785    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591789    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591794    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591798    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591802    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591806    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591810    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591814    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591821    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591825    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.603689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591830    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591835    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591839    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591843    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591847    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591851    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591855    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591859    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591864    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591887    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591894    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591900    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591905    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591909    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591913    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591917    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591921    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591925    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591929    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591933    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.604208 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591940    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591945    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591950    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591954    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591958    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591963    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591967    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591971    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591975    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591979    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591985    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591990    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.591995    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.592041    4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592105    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592113    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.604862 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592120    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592124    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592129    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592134    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592138    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592142    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592146    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592150    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592154    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592158    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592162    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592166    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592170    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592174    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592179    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592183    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592190    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592194    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592199    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592203    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.605478 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592207    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592211    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592216    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592220    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592224    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592228    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592232    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592238    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592242    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592246    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592253    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592257    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592261    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592265    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592269    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592273    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592278    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592282    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592286    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592289    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.606026 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592293    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592298    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592302    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592306    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592310    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592314    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592318    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592322    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592328    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592333    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592339    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592345    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592350    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592354    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592359    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592363    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592367    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592371    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592376    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.606523 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592380    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592384    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592391    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592395    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592400    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592404    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592408    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592412    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592417    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592421    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592425    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592429    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592433    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592437    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592443    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592449    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592454    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592458    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592462    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.607006 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592466    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592470    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592477    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592481    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592485    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592489    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592494    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592499    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592508    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592512    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592516    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592520    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592524    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592529    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592533    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592537    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592541    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592545    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592556    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592560    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.607455 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592565    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592569    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592573    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592577    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592581    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.592587    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.596409    4493 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.596424    4493 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.596443    4493 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.596448    4493 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596509    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596514    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596518    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596520    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596523    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596526    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596528    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596531    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596535    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596537    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.607993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596540    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596542    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596545    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596549    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596553    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596556    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596560    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596565    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596568    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596571    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596574    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596577    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596580    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596582    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596585    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596588    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596591    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596593    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596596    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.608518 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596599    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596601    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596604    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596606    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596609    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596612    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596614    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596617    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596620    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596622    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596625    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596627    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596630    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596632    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596635    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596637    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596640    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596642    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596645    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596647    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.609086 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596650    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596652    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596655    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596657    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596660    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596662    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596665    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596667    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596670    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596673    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596676    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596678    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596680    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596683    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596686    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596688    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596691    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596693    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596695    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.609574 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596698    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596700    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596702    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596705    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596708    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596711    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596713    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596715    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596718    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596720    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596723    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596725    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596728    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596730    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596733    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596735    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596738    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596740    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596743    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596745    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.610146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596748    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596750    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596753    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596755    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596758    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596760    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596762    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596765    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596767    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596770    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596773    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596775    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596777    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596780    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596782    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596784    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596787    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.610729 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596789    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.596815    4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596986    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596991    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596994    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.596997    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597000    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597003    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597005    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597008    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597011    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597013    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597016    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597018    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597021    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597024    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.611279 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597027    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597029    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597032    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597035    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597037    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597040    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597042    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597045    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597047    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597049    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597052    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597054    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597057    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597059    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597062    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597064    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597067    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597069    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597072    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.611802 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597074    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597076    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597079    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597081    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597084    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597086    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597089    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597091    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597094    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597097    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597099    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597102    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597105    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597108    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597110    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597113    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597116    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597118    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597121    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.612417 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597123    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597126    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597128    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597131    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597133    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597136    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597138    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597141    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597143    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597147    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597151    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597154    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597157    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597159    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597162    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597164    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597166    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597169    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597172    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597174    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.612990 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597176    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597179    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597181    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597184    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597186    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597188    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597192    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597198    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597201    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597204    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597206    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597209    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597211    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597213    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597216    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597220    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597223    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597226    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597228    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.613462 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597231    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597233    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597236    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597238    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597241    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597243    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597246    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597248    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597250    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597253    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597255    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597258    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597260    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597263    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597265    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.597288    4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 17:33:40.614004 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597330    4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597335    4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597338    4493 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597341    4493 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597345    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597350    4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597354    4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597358    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597369    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597372    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597375    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597378    4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597380    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597383    4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597385    4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597388    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597390    4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597393    4493 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597395    4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597398    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 17:33:40.614404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597400    4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597402    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597405    4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597407    4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597410    4493 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597413    4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597415    4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597418    4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597420    4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597422    4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597425    4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597427    4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597431    4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597434    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597437    4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597440    4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597442    4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597447    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597450    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 17:33:40.615169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597452    4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597455    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597458    4493 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597460    4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597462    4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597465    4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597468    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597470    4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597473    4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597475    4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597477    4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597480    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597482    4493 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597485    4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597487    4493 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597490    4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597492    4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597495    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597497    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597500    4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 17:33:40.615774 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597502    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597505    4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597507    4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597510    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597512    4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597516    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597519    4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597521    4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597523    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597526    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597530    4493 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597532    4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597535    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597538    4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597540    4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597543    4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597546    4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597549    4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597551    4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 17:33:40.616265 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597554    4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597556    4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597559    4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597561    4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597564    4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597567    4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597570    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597572    4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597574    4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597577    4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597580    4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597582    4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597585    4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597587    4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597590    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597592    4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597594    4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597597    4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597601    4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 17:33:40.616723 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597603    4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597606    4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597608    4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597611    4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597615    4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597617    4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597620    4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597622    4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:40.597626    4493 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.597673    4493 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.598664    4493 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.598716    4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.598906    4493 server.go:1014] "Starting client certificate rotation"
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.599044    4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.599096    4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:27:59 +0000 UTC" deadline="2028-08-17 09:30:42.194151164 +0000 UTC"
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.599132    4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15663h57m1.595022122s"
Nov 03 17:33:40.617225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.599426    4493 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:33:40.617607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.600822    4493 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 17:33:40.617607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.603149    4493 log.go:25] "Validated CRI v1 runtime API"
Nov 03 17:33:40.617607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.604800    4493 log.go:25] "Validated CRI v1 image API"
Nov 03 17:33:40.617607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.605593    4493 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 17:33:40.617607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.607233    4493 fs.go:135] Filesystem UUIDs: map[1cab9925-e0a1-4854-b491-2b24c98b6fb2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7a419278-b02a-4d76-b90d-b2e8e3d9846e:/dev/nvme0n1p4]
Nov 03 17:33:40.617816 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.607253    4493 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3445fdb14af949fee6f44944ec6c2423afc01bf99cafb7253e5737ad58d0b29a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3445fdb14af949fee6f44944ec6c2423afc01bf99cafb7253e5737ad58d0b29a/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36ecfe7b235090f1a3154276fb9d45099a4d2a1b3a4b78ab6283fd20c550ef2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36ecfe7b235090f1a3154276fb9d45099a4d2a1b3a4b78ab6283fd20c550ef2b/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42e50590bf2ca76e52dcd453d3f6036937af7061297cd967de8296b60d4c772c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42e50590bf2ca76e52dcd453d3f6036937af7061297cd967de8296b60d4c772c/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/473ffa27f4d4206455c8e944d4ee676dcf7c1786073575e82db66ead79e62989/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/473ffa27f4d4206455c8e944d4ee676dcf7c1786073575e82db66ead79e62989/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bf56a416a86b91ee9ee9c5473367cf25ea649e42797f5486653dbef49f71b82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bf56a416a86b91ee9ee9c5473367cf25ea649e42797f5486653dbef49f71b82/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d269573568f76713c6eefcebcd2d01ee83ec9228ce3677b68c83ea0a3521991/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d269573568f76713c6eefcebcd2d01ee83ec9228ce3677b68c83ea0a3521991/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d4dbe0346f4bbd212c9bb75aef2d3a2661ea25537ad136bbc300f14477518fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d4dbe0346f4bbd212c9bb75aef2d3a2661ea25537ad136bbc300f14477518fc/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70ebd968fb3a7421674a5187a2d775e941937203b782036231e36027ef62143c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70ebd968fb3a7421674a5187a2d775e941937203b782036231e36027ef62143c/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c685e7e7c4ae9e5c08c3a4572ccead9e81a17d3f49144c7d75906a61c876ce2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c685e7e7c4ae9e5c08c3a4572ccead9e81a17d3f49144c7d75906a61c876ce2/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd82c239c67a9360494a131b1ec0674a040a585daf4235ebd55469843584b23e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd82c239c67a9360494a131b1ec0674a040a585daf4235ebd55469843584b23e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f06fe48aa384d0f6d9876177748c36405e6e4845ae0ae70efde4e2c92053636f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f06fe48aa384d0f6d9876177748c36405e6e4845ae0ae70efde4e2c92053636f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f396f99503ac8aef8cf922b88db3e418127442b9ccd4a9ec0d5d0e67ad871675/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f396f99503ac8aef8cf922b88db3e418127442b9ccd4a9ec0d5d0e67ad871675/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f605902602a8dfb9a6c920440583f3e453906d8a8bfe5ab5aa0d8c252c0ab0a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f605902602a8dfb9a6c920440583f3e453906d8a8bfe5ab5aa0d8c252c0ab0a3/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01b8876a-2097-40ee-9b09-4ec46606a7b1/volumes/kubernetes.io~projected/kube-api-access-2th75:{mountpoint:/var/lib/kubelet/pods/01b8876a-2097-40ee-9b09-4ec46606a7b1/volumes/kubernetes.io~projected/kube-api-access-2th75 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1758fefb-559e-433d-a077-bb4b574853e1/volumes/kubernetes.io~projected/kube-api-access-lbnnd:{mountpoint:/var/lib/kubelet/pods/1758fefb-559e-433d-a077-bb4b574853e1/volumes/kubernetes.io~projected/kube-api-access-lbnnd major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1758fefb-559e-433d-a077-bb4b574853e1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1758fefb-559e-433d-a077-bb4b574853e1/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e373a48-c67d-4cd3-8837-166a41802085/volumes/kubernetes.io~projected/kube-api-access-p2kb4:{mountpoint:/var/lib/kubelet/pods/1e373a48-c67d-4cd3-8837-166a41802085/volumes/kubernetes.io~projected/kube-api-access-p2kb4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c073539-dddd-4fd2-88ee-52fc61b9f50d/volumes/kubernetes.io~projected/kube-api-access-2pnh6:{mountpoint:/var/lib/kubelet/pods/2c073539-dddd-4fd2-88ee-52fc61b9f50d/volumes/kubernetes.io~projected/kube-api-access-2pnh6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44f3aa72-92ab-4dff-9c6a-2c78626e8b50/volumes/kubernetes.io~projected/kube-api-access-v5czc:{mountpoint:/var/lib/kubelet/pods/44f3aa72-92ab-4dff-9c6a-2c78626e8b50/volumes/kubernetes.io~projected/kube-api-access-v5czc major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60cede5f-b376-45dd-b774-feb3598cf479/volumes/kubernetes.io~projected/kube-api-access-56t79:{mountpoint:/var/lib/kubelet/pods/60cede5f-b376-45dd-b774-feb3598cf479/volumes/kubernetes.io~projected/kube-api-access-56t79 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volumes/kubernetes.io~projected/kube-api-access-5km7g:{mountpoint:/var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volumes/kubernetes.io~projected/kube-api-access-5km7g major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8222297a-ec44-495a-b6bd-6b58501aadd4/volumes/kubernetes.io~projected/kube-api-access-44kmt:{mountpoint:/var/lib/kubelet/pods/8222297a-ec44-495a-b6bd-6b58501aadd4/volumes/kubernetes.io~projected/kube-api-access-44kmt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/944b1b52-5928-4c8d-9d19-f6a17c45f9e7/volumes/kubernetes.io~projected/kube-api-access-458s7:{mountpoint:/var/lib/kubelet/pods/944b1b52-5928-4c8d-9d19-f6a17c45f9e7/volumes/kubernetes.io~projected/kube-api-access-458s7 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9699d94f-597e-4612-a469-3e657bfc9a55/volumes/kubernetes.io~projected/kube-api-access-nm67z:{mountpoint:/var/lib/kubelet/pods/9699d94f-597e-4612-a469-3e657bfc9a55/volumes/kubernetes.io~projected/kube-api-access-nm67z major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9699d94f-597e-4612-a469-3e657bfc9a55/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/9699d94f-597e-4612-a469-3e657bfc9a55/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b82e04a-d3d2-43de-ad90-abccdeb48e7e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9b82e04a-d3d2-43de-ad90-abccdeb48e7e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c42f2ccd-9e73-4f22-8370-409ca511aeb3/volumes/kubernetes.io~projected/kube-api-access-pstsl:{mountpoint:/var/lib/kubelet/pods/c42f2ccd-9e73-4f22-8370-409ca511aeb3/volumes/kubernetes.io~projected/kube-api-access-pstsl major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~projected/kube-api-access-pz9pr:{mountpoint:/var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~projected/kube-api-access-pz9pr major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c85b087b-5e24-45d0-abe3-d8e72dbbb74e/volumes/kubernetes.io~projected/kube-api-access-sdj6s:{mountpoint:/var/lib/kubelet/pods/c85b087b-5e24-45d0-abe3-d8e72dbbb74e/volumes/kubernetes.io~projected/kube-api-access-sdj6s major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3138bcb-bcf2-49bc-959d-c2177605f704/volumes/kubernetes.io~projected/kube-api-access-g4vjp:{mountpoint:/var/lib/kubelet/pods/d3138bcb-bcf2-49bc-959d-c2177605f704/volumes/kubernetes.io~projected/kube-api-access-g4vjp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9/volumes/kubernetes.io~projected/kube-api-access-b5l8w:{mountpoint:/var/lib/kubelet/pods/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9/volumes/kubernetes.io~projected/kube-api-access-b5l8w major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40/volumes/kubernetes.io~projected/kube-api-access-lbtkn:{mountpoint:/var/lib/kubelet/pods/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40/volumes/kubernetes.io~projected/kube-api-access-lbtkn major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/cf4c97f450b3950b6fdbfcce43489303266525e779131d3c9e79bb0910f7812c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1fd8314929d9748b118752560cf406149b8bcdf5c03fcad0cb0c366847e2afaa/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/82d26971fe6913fd19cdaa99f0f15fb1577edd6bebc0920af1f4f8321808e275/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b20e1620d6f4a9240d77e8d8c4eab2a756c8df5a2dad079c999b1dd8329f359b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ad7e2b148bdbff7e0868bf1ede6138cb03cc6f8f5747b7791f1a5f6fd6139edc/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0f61ddf05effa1aaa31a766bef8363fd23c2043fcff2f038d2b4f7f25d0d41dd/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4abcaa08ddb7421dd13cf0c82e2d5d0537493ae74ed89a0b138ae40ee485fecb/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3c0888b040dc31c220fe99bfb6ba9bdce97a70b3dc8dd791dda7904a590bf5a3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d3ee9fcc778fea5c306e011fe3449d6b4d0450dee8094e382ca7757c8753dc74/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/beaf1f5871cc96379e0e3bfeff814e4ce7408f5f063c38aad6dabaae767c5581/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/6584c1a5980b8646834e7e6d04eb53cad5166c633da707c16d288b2918cb6639/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/2af8e943f6287d9f5c52afac6a5128be093502a591b745cc60bc8de2822cb49e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ffedda7bce75df80942325d9391659fbec32319b9e643de082d36499c739b013/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c83377a9b5afa1ec4885c7ff3aef34b8c3fa49f0900da7bfd26d158298fdfe8a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ba739b7e4ec287df9cbdc4a54857504e83c98568845c26fac093ba2b85b59e43/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7c44db7626dd6424180f81ce8de481cab665a32560de76d1378f9e9ad5cfc65f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a1d4884b673f2ca7f7b40a582e83e238e299ea7f8df1b4ef4617c595d7d96cff/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/813bed900c99bea26a1e6d81f173c270b5b48ad90f8e78054c1d0200ce7fca6b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/301bb521d89f3ecd0adf3786f6a18c665007e1c041d7f6f723c36431d5bb0292/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/17921715395cc972f4f0e00dea7e2c73e9d6eeeff1f4e7843b0cf1526f4d04c4/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f187ae46550e0e958b96999d3cd34a1bc9f52401f1065073b13fc7eb5726530c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/03f7e10a68de4b8da59398434919ed7473b10c57b607db1730ef8e7de32a9fe8/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/d1c846abefa39727f7d8b350ecaf4e729e653a6e8240973ae1f08371019d8680/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/221a91032c11c6248ef9f26430d9b90f2afb87474577802897dfdd8058584706/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/d76666d2151716a4611480998dc07f7277f8fc9988d48d54c8ea85960bab5c96/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/139130f03ce93df6139eabae8b232b5af078635f29c7b1a194ed7753e6988b66/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/735e30465a3b1735bfd1af214096a3fbd33f7e1df1a71ac2dd6c370f056dccf6/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/d8942f26353082dca625d802cad5697ffd6e08ee87615abbf8e9c25508b156a0/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/9d609e7b40b4c7944ffdd13085e7deb902d7961f67c73feb873693f98e1580e5/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/aa74c63c269586745ed5ed9d30d3e832488ae091352b360821ca950dc31341a1/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/5fe2c20ddc627d0563b2101d63adbf74019f812d0b89c61a3700fbf3108a8afe/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a5ef51d1bc49619051d0c1d60a1297caf4bbf67022d8277346e11cb36fd95246/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/938e589e25cb9a09451d98cfea0938213151af7495466094bada71dac4251e30/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/83bd62c1bf6333d97a44b4b6992af3a7dcff3571c6bb9e6258055490a06f8477/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f64e016158279af445542aeded93c38b6136a33b415e87cc3b5c211192d8eb0b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2995b00d416304c81733e61e1215785c6f5d49f211281b150552e8b88d74c5ae/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/10114141458a8bdb89d1035ceea71bb8654237b5249cefd57b035377b9f3fb48/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/73f77530b2537ca1c1f3f26e5467cb8eb2063899d6e473712c760cf2d7327f58/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c6c6f8e0ba04136bf9ee8bf536e498d931bc65cb44f09e65d1a2fe62388d4085/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ab4f1a0e140adf21824c89396b3961cc749ca3b9af3d99a8593a47529ee88562/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.613288    4493 manager.go:217] Machine: {Timestamp:2026-11-03 17:33:40.612817411 +0000 UTC m=+0.071988387 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100082 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d2d378e1e48920c2567d358c4561 SystemUUID:ec27d2d3-78e1-e489-20c2-567d358c4561 BootID:0c63f996-05c6-47c2-bd3e-6b62216945a3 Filesystems:[{Device:/var/lib/kubelet/pods/01b8876a-2097-40ee-9b09-4ec46606a7b1/volumes/kubernetes.io~projected/kube-api-access-2th75 DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3445fdb14af949fee6f44944ec6c2423afc01bf99cafb7253e5737ad58d0b29a/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volumes/kubernetes.io~projected/kube-api-access-5km7g DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44f3aa72-92ab-4dff-9c6a-2c78626e8b50/volumes/kubernetes.io~projected/kube-api-access-v5czc DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d3138bcb-bcf2-49bc-959d-c2177605f704/volumes/kubernetes.io~projected/kube-api-access-g4vjp DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f06fe48aa384d0f6d9876177748c36405e6e4845ae0ae70efde4e2c92053636f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bf56a416a86b91ee9ee9c5473367cf25ea649e42797f5486653dbef49f71b82/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b82e04a-d3d2-43de-ad90-abccdeb48e7e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c85b087b-5e24-45d0-abe3-d8e72dbbb74e/volumes/kubernetes.io~projected/kube-api-access-sdj6s DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d269573568f76713c6eefcebcd2d01ee83ec9228ce3677b68c83ea0a3521991/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1758fefb-559e-433d-a077-bb4b574853e1/volumes/kubernetes.io~projected/kube-api-access-lbnnd DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd82c239c67a9360494a131b1ec0674a040a585daf4235ebd55469843584b23e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c42f2ccd-9e73-4f22-8370-409ca511aeb3/volumes/kubernetes.io~projected/kube-api-access-pstsl DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c685e7e7c4ae9e5c08c3a4572ccead9e81a17d3f49144c7d75906a61c876ce2/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9699d94f-597e-4612-a469-3e657bfc9a55/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40/volumes/kubernetes.io~projected/kube-api-access-lbtkn DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f396f99503ac8aef8cf922b88db3e418127442b9ccd4a9ec0d5d0e67ad871675/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36ecfe7b235090f1a3154276fb9d45099a4d2a1b3a4b78ab6283fd20c550ef2b/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60cede5f-b376-45dd-b774-feb3598cf479/volumes/kubernetes.io~projected/kube-api-access-56t79 DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c073539-dddd-4fd2-88ee-52fc61b9f50d/volumes/kubernetes.io~projected/kube-api-access-2pnh6 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42e50590bf2ca76e52dcd453d3f6036937af7061297cd967de8296b60d4c772c/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d4dbe0346f4bbd212c9bb75aef2d3a2661ea25537ad136bbc300f14477518fc/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9699d94f-597e-4612-a469-3e657bfc9a55/volumes/kubernetes.io~projected/kube-api-access-nm67z DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/8222297a-ec44-495a-b6bd-6b58501aadd4/volumes/kubernetes.io~projected/kube-api-access-44kmt DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/473ffa27f4d4206455c8e944d4ee676dcf7c1786073575e82db66ead79e62989/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f605902602a8dfb9a6c920440583f3e453906d8a8bfe5ab5aa0d8c252c0ab0a3/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70ebd968fb3a7421674a5187a2d775e941937203b782036231e36027ef62143c/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/944b1b52-5928-4c8d-9d19-f6a17c45f9e7/volumes/kubernetes.io~projected/kube-api-access-458s7 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09/volumes/kubernetes.io~projected/kube-api-access-pz9pr DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1e373a48-c67d-4cd3-8837-166a41802085/volumes/kubernetes.io~projected/kube-api-access-p2kb4 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1758fefb-559e-433d-a077-bb4b574853e1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9/volumes/kubernetes.io~projected/kube-api-access-b5l8w DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3445fdb14af949f MacAddress:5a:3c:f1:3e:5b:c7 Speed:10000 Mtu:8901} {Name:36ecfe7b235090f MacAddress:9e:14:52:eb:06:35 Speed:10000 Mtu:8901} {Name:62a07abd2e0db07 MacAddress:ae:cc:51:d4:1e:8f Speed:10000 Mtu:8901} {Name:6d4dbe0346f4bbd MacAddress:1a:f0:47:ce:0d:ca Speed:10000 Mtu:8901} {Name:70ebd968fb3a742 MacAddress:02:f4:16:87:15:20 Speed:10000 Mtu:8901} {Name:8c685e7e7c4ae9e MacAddress:1e:83:5e:37:f9:07 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:0f:cf:1a:9d:05 Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:9e:44:a5:c9:18 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:0f:cf:1a:9d:05 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8e:ac:1a:be:87:9e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:dd:ac:62:7d:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.613535    4493 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.613573    4493 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.613832    4493 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.613850    4493 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-172.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.613992    4493 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614001    4493 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614011    4493 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614031    4493 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614115    4493 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614309    4493 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614405    4493 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614437    4493 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614449    4493 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614459    4493 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614468    4493 kubelet.go:405] "Adding apiserver pod source"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.614476    4493 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.615023    4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.615032    4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:33:40.618067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.615434    4493 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 17:33:40.665130 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665112    4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 17:33:40.665130 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665128    4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 17:33:40.665250 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665236    4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 17:33:40.665448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665436    4493 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 17:33:40.665483 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665466    4493 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 17:33:40.665576 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665567    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 17:33:40.665615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665582    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 17:33:40.665615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665589    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 17:33:40.665615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665595    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 17:33:40.665615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665600    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 17:33:40.665615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665606    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 17:33:40.665615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665612    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 17:33:40.665853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665618    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 17:33:40.665853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665625    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 17:33:40.665853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665631    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 17:33:40.665853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665640    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 17:33:40.665853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665650    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 17:33:40.665853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665675    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 17:33:40.665853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.665680    4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 17:33:40.669116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.669099    4493 server.go:1263] "Started kubelet"
Nov 03 17:33:40.669290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.669254    4493 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 17:33:40.669370 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.669248    4493 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 17:33:40.669370 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.669334    4493 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 17:33:40.669761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.669671    4493 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 17:33:40.669706 ip-10-0-131-172 systemd[1]: Started Kubernetes Kubelet.
Nov 03 17:33:40.670694 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.670677    4493 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 17:33:40.672861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.672844    4493 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 17:33:40.672956 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.672861    4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 17:33:40.673012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.672939    4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:01 +0000 UTC" deadline="2028-04-11 19:15:17.931951896 +0000 UTC"
Nov 03 17:33:40.673012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.672971    4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12601h41m37.258984084s"
Nov 03 17:33:40.673112 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.673039    4493 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:33:40.673112 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.673043    4493 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 17:33:40.673112 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.673055    4493 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 17:33:40.673112 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.673058    4493 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 17:33:40.674059 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.674026    4493 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 17:33:40.674274 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.674147    4493 factory.go:55] Registering systemd factory
Nov 03 17:33:40.674274 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.674160    4493 factory.go:223] Registration of the systemd container factory successfully
Nov 03 17:33:40.675395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.675375    4493 factory.go:153] Registering CRI-O factory
Nov 03 17:33:40.675395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.675398    4493 factory.go:223] Registration of the crio container factory successfully
Nov 03 17:33:40.675542 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.675444    4493 factory.go:103] Registering Raw factory
Nov 03 17:33:40.675542 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.675462    4493 manager.go:1199] Started watching for new ooms in manager
Nov 03 17:33:40.676522 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.676267    4493 manager.go:319] Starting recovery of all containers
Nov 03 17:33:40.676924 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.676890    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8222297a-ec44-495a-b6bd-6b58501aadd4" volumeName="kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 17:33:40.677007 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.676944    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a0388c-1177-45a8-8eb8-92ef23d8f0bf" volumeName="kubernetes.io/configmap/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-konnectivity-ca" seLinuxMountContext=""
Nov 03 17:33:40.677007 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.676976    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c42f2ccd-9e73-4f22-8370-409ca511aeb3" volumeName="kubernetes.io/configmap/c42f2ccd-9e73-4f22-8370-409ca511aeb3-config-volume" seLinuxMountContext=""
Nov 03 17:33:40.677007 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.676994    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" volumeName="kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w" seLinuxMountContext=""
Nov 03 17:33:40.677185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677044    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01b8876a-2097-40ee-9b09-4ec46606a7b1" volumeName="kubernetes.io/projected/01b8876a-2097-40ee-9b09-4ec46606a7b1-kube-api-access-2th75" seLinuxMountContext=""
Nov 03 17:33:40.677185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677072    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8222297a-ec44-495a-b6bd-6b58501aadd4" volumeName="kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:33:40.677185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677087    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c42f2ccd-9e73-4f22-8370-409ca511aeb3" volumeName="kubernetes.io/projected/c42f2ccd-9e73-4f22-8370-409ca511aeb3-kube-api-access-pstsl" seLinuxMountContext=""
Nov 03 17:33:40.677185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677117    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef771bc1-ab90-44d0-93a5-5e3fa2a8db40" volumeName="kubernetes.io/projected/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-kube-api-access-lbtkn" seLinuxMountContext=""
Nov 03 17:33:40.677185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677139    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9699d94f-597e-4612-a469-3e657bfc9a55" volumeName="kubernetes.io/configmap/9699d94f-597e-4612-a469-3e657bfc9a55-signing-cabundle" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677181    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9699d94f-597e-4612-a469-3e657bfc9a55" volumeName="kubernetes.io/projected/9699d94f-597e-4612-a469-3e657bfc9a55-kube-api-access-nm67z" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677255    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01b8876a-2097-40ee-9b09-4ec46606a7b1" volumeName="kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-cni-binary-copy" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677268    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c073539-dddd-4fd2-88ee-52fc61b9f50d" volumeName="kubernetes.io/projected/2c073539-dddd-4fd2-88ee-52fc61b9f50d-kube-api-access-2pnh6" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677282    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c42f2ccd-9e73-4f22-8370-409ca511aeb3" volumeName="kubernetes.io/empty-dir/c42f2ccd-9e73-4f22-8370-409ca511aeb3-tmp-dir" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677316    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef771bc1-ab90-44d0-93a5-5e3fa2a8db40" volumeName="kubernetes.io/configmap/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-iptables-alerter-script" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677338    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01b8876a-2097-40ee-9b09-4ec46606a7b1" volumeName="kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-daemon-config" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677353    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1758fefb-559e-433d-a077-bb4b574853e1" volumeName="kubernetes.io/projected/1758fefb-559e-433d-a077-bb4b574853e1-kube-api-access-lbnnd" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677402    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1758fefb-559e-433d-a077-bb4b574853e1" volumeName="kubernetes.io/secret/1758fefb-559e-433d-a077-bb4b574853e1-cert" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677416    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c37bc06-03be-48f6-9bd9-f6eb3e711ef0" volumeName="kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677469    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8222297a-ec44-495a-b6bd-6b58501aadd4" volumeName="kubernetes.io/projected/8222297a-ec44-495a-b6bd-6b58501aadd4-kube-api-access-44kmt" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677493    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a0388c-1177-45a8-8eb8-92ef23d8f0bf" volumeName="kubernetes.io/secret/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-agent-certs" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677504    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3138bcb-bcf2-49bc-959d-c2177605f704" volumeName="kubernetes.io/projected/d3138bcb-bcf2-49bc-959d-c2177605f704-kube-api-access-g4vjp" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677514    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" volumeName="kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677582    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c37bc06-03be-48f6-9bd9-f6eb3e711ef0" volumeName="kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-env-overrides" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677614    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8222297a-ec44-495a-b6bd-6b58501aadd4" volumeName="kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677626    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b82e04a-d3d2-43de-ad90-abccdeb48e7e" volumeName="kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677653    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5cc09b0-ed27-4664-9d61-81d7e3f7ef09" volumeName="kubernetes.io/projected/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-kube-api-access-pz9pr" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677681    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c85b087b-5e24-45d0-abe3-d8e72dbbb74e" volumeName="kubernetes.io/projected/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-kube-api-access-sdj6s" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677692    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3138bcb-bcf2-49bc-959d-c2177605f704" volumeName="kubernetes.io/configmap/d3138bcb-bcf2-49bc-959d-c2177605f704-serviceca" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677727    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" volumeName="kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677739    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c073539-dddd-4fd2-88ee-52fc61b9f50d" volumeName="kubernetes.io/empty-dir/2c073539-dddd-4fd2-88ee-52fc61b9f50d-tmp-dir" seLinuxMountContext=""
Nov 03 17:33:40.677741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677753    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60cede5f-b376-45dd-b774-feb3598cf479" volumeName="kubernetes.io/projected/60cede5f-b376-45dd-b774-feb3598cf479-kube-api-access-56t79" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677778    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c37bc06-03be-48f6-9bd9-f6eb3e711ef0" volumeName="kubernetes.io/projected/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-kube-api-access-5km7g" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677825    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944b1b52-5928-4c8d-9d19-f6a17c45f9e7" volumeName="kubernetes.io/empty-dir/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-data-volume" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677836    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5cc09b0-ed27-4664-9d61-81d7e3f7ef09" volumeName="kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-tuned" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677867    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5cc09b0-ed27-4664-9d61-81d7e3f7ef09" volumeName="kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-tmp" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677897    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e373a48-c67d-4cd3-8837-166a41802085" volumeName="kubernetes.io/projected/1e373a48-c67d-4cd3-8837-166a41802085-kube-api-access-p2kb4" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.677991    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44f3aa72-92ab-4dff-9c6a-2c78626e8b50" volumeName="kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.678074    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c37bc06-03be-48f6-9bd9-f6eb3e711ef0" volumeName="kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-config" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.678082    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944b1b52-5928-4c8d-9d19-f6a17c45f9e7" volumeName="kubernetes.io/projected/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-api-access-458s7" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.678095    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9699d94f-597e-4612-a469-3e657bfc9a55" volumeName="kubernetes.io/secret/9699d94f-597e-4612-a469-3e657bfc9a55-signing-key" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.678139    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" volumeName="kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.678148    4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c37bc06-03be-48f6-9bd9-f6eb3e711ef0" volumeName="kubernetes.io/secret/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.678163    4493 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 17:33:40.678802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.678170    4493 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 17:33:40.701700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.701670    4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 17:33:40.704839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.704808    4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 17:33:40.704839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.704838    4493 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 17:33:40.705003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.704861    4493 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 17:33:40.705261 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.705237    4493 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 17:33:40.728042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.727963    4493 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="f5a11f2da2695f93a3ffc769c566ab98b505e0310afcffefe28e016fd56e8bc5" exitCode=0
Nov 03 17:33:40.728042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.727993    4493 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="da659913732b37413354e027110ab62c2ff043bb5fd463b5645a134fa18c41c7" exitCode=0
Nov 03 17:33:40.728042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.728008    4493 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="62d6687be3567cbd932cfaa54b357987c9d141e748a2ef352794284c8766b204" exitCode=0
Nov 03 17:33:40.728042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.728016    4493 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="890fd62c93f37e68acd7b933ccd3dba0bc23600519387acfe980b11c1b819ed6" exitCode=0
Nov 03 17:33:40.728042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.728024    4493 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="41aaf4fbd1bb0f5081f95044019f870837a49ee3557eb4176a0f66bb7f5f7590" exitCode=0
Nov 03 17:33:40.728042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.728029    4493 generic.go:356] "Generic (PLEG): container finished" podID="8222297a-ec44-495a-b6bd-6b58501aadd4" containerID="d4a5a7de8a4b1700961d06f59bf7caf5aabe2f005eaf9ce1217067bc7e01ba50" exitCode=0
Nov 03 17:33:40.730430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.730397    4493 generic.go:356] "Generic (PLEG): container finished" podID="dbe46139cd173525f608561351491180" containerID="621c5a17322dea1cd41e9e06c2d294287980497158ac977991d28d572613e7b4" exitCode=0
Nov 03 17:33:40.746481 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.746461    4493 manager.go:324] Recovery completed
Nov 03 17:33:40.761343 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.761319    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:40.762537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.762519    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:40.762626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.762555    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:40.762626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.762569    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:40.763949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.763926    4493 cpu_manager.go:226] "Starting" policy="none"
Nov 03 17:33:40.763949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.763943    4493 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 17:33:40.763949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.763947    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:40.764088 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.763963    4493 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 17:33:40.764127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764112    4493 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 17:33:40.764157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764125    4493 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 17:33:40.764157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764141    4493 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 17:33:40.764157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764147    4493 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 17:33:40.764157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764152    4493 policy_none.go:50] "Start"
Nov 03 17:33:40.764268 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764160    4493 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 17:33:40.764268 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764170    4493 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 17:33:40.764320 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764306    4493 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 17:33:40.764320 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764314    4493 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 17:33:40.764320 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764319    4493 policy_none.go:44] "Start"
Nov 03 17:33:40.764715 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764700    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:40.764782 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764727    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:40.764782 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.764737    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:40.768170 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.768148    4493 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 17:33:40.770697 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.770685    4493 manager.go:341] "Starting Device Plugin manager"
Nov 03 17:33:40.770749 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.770707    4493 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 17:33:40.770749 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.770717    4493 server.go:87] "Starting device plugin registration server"
Nov 03 17:33:40.771040 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.771030    4493 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 17:33:40.771072 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.771042    4493 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 17:33:40.771163 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.771142    4493 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 17:33:40.771309 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.771267    4493 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 17:33:40.771368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.771319    4493 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 17:33:40.771368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.771327    4493 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 17:33:40.771491 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.771402    4493 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:33:40.771307628Z","UUID":"b8ac3483-b5e6-4d08-a03e-0ab1a196b361","Handler":null,"Name":"","Endpoint":""}
Nov 03 17:33:40.772714 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.772692    4493 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 17:33:40.772811 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.772736    4493 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-172.us-west-1.compute.internal\" not found"
Nov 03 17:33:40.772957 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.772933    4493 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 17:33:40.773023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.772969    4493 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 17:33:40.806175 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.806128    4493 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"]
Nov 03 17:33:40.806300 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.806266    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:40.807106 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.807090    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:40.807176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.807119    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:40.807176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.807130    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:40.807364 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.807354    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:40.807505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.807484    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:40.808080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808057    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:40.808080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808067    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:40.808203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808087    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:40.808203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808100    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:40.808203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808088    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:40.808203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808165    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:40.808322 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.808240    4493 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.808460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808425    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"dbe46139cd173525f608561351491180","Type":"ContainerStarted","Data":"f48ee5c96cdb0e87d3cb5769dd186b2dda1ad614ce93b6ade3691ddba20c0ffb"}
Nov 03 17:33:40.808489 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808472    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"dbe46139cd173525f608561351491180","Type":"ContainerDied","Data":"621c5a17322dea1cd41e9e06c2d294287980497158ac977991d28d572613e7b4"}
Nov 03 17:33:40.808521 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808493    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"dbe46139cd173525f608561351491180","Type":"ContainerStarted","Data":"d3e48d609035acf73f4a6f7e62b77e1a6b5db90100706dc1cee130dbc75e3439"}
Nov 03 17:33:40.808552 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808527    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:40.808603 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808527    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"3eefabd2e362fe1b3609803d32f993c7","Type":"ContainerStarted","Data":"cab66d4025d5649fda1a592e56e38df069679cbd8e2f4725af23f061deec71a6"}
Nov 03 17:33:40.808635 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.808624    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal" event={"ID":"3eefabd2e362fe1b3609803d32f993c7","Type":"ContainerStarted","Data":"8414db50ed6b084c9bfed3b0f64ea47cca71d2003fb61b126915dbebf3d25bd4"}
Nov 03 17:33:40.809191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.809178    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:40.809249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.809199    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:40.809249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.809209    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:40.809345 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.809331    4493 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-172.us-west-1.compute.internal\" not found" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.829970 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.829951    4493 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:40.830043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.830029    4493 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 17:33:40.831520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.831497    4493 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:40.831600 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.831531    4493 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:40.871652 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.871627    4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 17:33:40.872502 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.872481    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 17:33:40.872584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.872512    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 17:33:40.872584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.872527    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 17:33:40.872584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.872568    4493 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.873651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.873636    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.882538 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.882515    4493 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 17:33:40.930430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.930404    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3eefabd2e362fe1b3609803d32f993c7-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"3eefabd2e362fe1b3609803d32f993c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.930531 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.930432    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.930531 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.930451    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.935333 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.935316    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:33:40.935391 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.935363    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.935391 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.935376    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.987805 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.987739    4493 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.987805 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.987802    4493 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:40.988383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.988361    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeNotReady"
Nov 03 17:33:40.988484 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.988396    4493 setters.go:546] "Node became not ready" node="ip-10-0-131-172.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T17:33:40Z","lastTransitionTime":"2026-11-03T17:33:40Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 17:33:40.998216 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:40.998197    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:33:40.998281 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:40.998239    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.003619 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.003595    4493 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-172.us-west-1.compute.internal" is owned by different node (oldNodeID="a4e7ed0f-8df0-4d92-a0a1-5a4f8a65f025", newNodeID=""), cleaning up...
Nov 03 17:33:41.030765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.030742    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3eefabd2e362fe1b3609803d32f993c7-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"3eefabd2e362fe1b3609803d32f993c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.030862 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.030779    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.030862 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.030797    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.030862 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.030824    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3eefabd2e362fe1b3609803d32f993c7-config\") pod \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"3eefabd2e362fe1b3609803d32f993c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.030862 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.030834    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.031034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.030841    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbe46139cd173525f608561351491180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" (UID: \"dbe46139cd173525f608561351491180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.615409 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.615367    4493 apiserver.go:52] "Watching apiserver"
Nov 03 17:33:41.724564 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.724535    4493 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 17:33:41.725005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.724987    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl","openshift-multus/multus-7qrt7","openshift-insights/insights-runtime-extractor-82x2r","openshift-multus/network-metrics-daemon-2ccdq","openshift-network-diagnostics/network-check-target-7v8km","openshift-dns/node-resolver-hcf4k","kube-system/global-pull-secret-syncer-648bg","kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-rgmgf","openshift-multus/multus-additional-cni-plugins-vfd7n","openshift-service-ca/service-ca-5fffc97bdf-gjgn4","openshift-dns/dns-default-4w9qr","openshift-network-operator/iptables-alerter-88fkm","kube-system/konnectivity-agent-t4sns","openshift-ingress-canary/ingress-canary-6shfz","openshift-insights/periodic-gathering-fssvd-2gvm4","openshift-cluster-node-tuning-operator/tuned-vcxvs","openshift-image-registry/node-ca-9n9mf","openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"]
Nov 03 17:33:41.726129 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.726109    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:41.729855 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.729832    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.730947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.730924    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.748688 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.748669    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.748783 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.748770    4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.774984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.774963    4493 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 17:33:41.791044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791023    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 17:33:41.791133 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791026    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxb7s\""
Nov 03 17:33:41.791239 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791219    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 17:33:41.791239 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791232    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfxsl\""
Nov 03 17:33:41.791440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791259    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xxxqk\""
Nov 03 17:33:41.791440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791261    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2qfzl\""
Nov 03 17:33:41.791440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791300    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zl7j\""
Nov 03 17:33:41.791440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791310    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-glkrd\""
Nov 03 17:33:41.791440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791314    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xhj6\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791452    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jxmj\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791505    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791523    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791564    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791568    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791570    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791584    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.791700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791609    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tvvh\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791835    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791849    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791899    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791919    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.791950    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792009    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792027    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792033    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.792083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792066    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792068    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792102    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792144    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792177    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792280    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nx9c7\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792318    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xhql4\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792336    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792352    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792288    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 17:33:41.792512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792376    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 17:33:41.792968 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.792952    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.835267 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835241    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f2ccd-9e73-4f22-8370-409ca511aeb3-tmp-dir\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.835340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835268    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1758fefb-559e-433d-a077-bb4b574853e1-cert\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:41.835340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835285    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnnd\" (UniqueName: \"kubernetes.io/projected/1758fefb-559e-433d-a077-bb4b574853e1-kube-api-access-lbnnd\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:41.835340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835312    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.835340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835328    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-var-lib-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.835523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835346    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-script-lib\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.835523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835370    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdj6s\" (UniqueName: \"kubernetes.io/projected/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-kube-api-access-sdj6s\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:41.835523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835404    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f2ccd-9e73-4f22-8370-409ca511aeb3-tmp-dir\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.835523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835418    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.835523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835471    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-multus-certs\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.835523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835496    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835586    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm67z\" (UniqueName: \"kubernetes.io/projected/9699d94f-597e-4612-a469-3e657bfc9a55-kube-api-access-nm67z\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835642    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-iptables-alerter-script\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835674    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-multus\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835710    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-script-lib\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835721    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-kubelet\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835750    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2th75\" (UniqueName: \"kubernetes.io/projected/01b8876a-2097-40ee-9b09-4ec46606a7b1-kube-api-access-2th75\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835782    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-systemd-units\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835816    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56t79\" (UniqueName: \"kubernetes.io/projected/60cede5f-b376-45dd-b774-feb3598cf479-kube-api-access-56t79\") pod \"migrator-66df5d496b-xjvqh\" (UID: \"60cede5f-b376-45dd-b774-feb3598cf479\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835834    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-binary-copy\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835843    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-conf\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.835906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835866    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-sys\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835917    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-host-slash\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835939    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-etc-kubernetes\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835970    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-iptables-alerter-script\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.835982    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vjp\" (UniqueName: \"kubernetes.io/projected/d3138bcb-bcf2-49bc-959d-c2177605f704-kube-api-access-g4vjp\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836012    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-kubelet\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836042    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5l8w\" (UniqueName: \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836085    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458s7\" (UniqueName: \"kubernetes.io/projected/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-api-access-458s7\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836146    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836243    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-system-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836279    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-cni-binary-copy\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836305    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-k8s-cni-cncf-io\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836390    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836422    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836451    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-device-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836474    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-modprobe-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836489    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836502    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-cnibin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836522    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-socket-dir-parent\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836543    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-slash\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836562    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-etc-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836577    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-node-log\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836591    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836635    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836639    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-cni-binary-copy\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836657    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9699d94f-597e-4612-a469-3e657bfc9a55-signing-key\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836673    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-daemon-config\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836799    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-bin\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836823    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.836824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836838    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c073539-dddd-4fd2-88ee-52fc61b9f50d-hosts-file\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836854    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c073539-dddd-4fd2-88ee-52fc61b9f50d-tmp-dir\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836863    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-daemon-config\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836888    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-lib-modules\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836930    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-tuned\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.836989    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9pr\" (UniqueName: \"kubernetes.io/projected/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-kube-api-access-pz9pr\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837009    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-tuned\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837024    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-hostroot\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837024    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c073539-dddd-4fd2-88ee-52fc61b9f50d-tmp-dir\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837035    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837046    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837075    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km7g\" (UniqueName: \"kubernetes.io/projected/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-kube-api-access-5km7g\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837171    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovn-node-metrics-cert\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837186    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-var-lib-kubelet\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837219    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-bin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837250    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtkn\" (UniqueName: \"kubernetes.io/projected/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-kube-api-access-lbtkn\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837274    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3138bcb-bcf2-49bc-959d-c2177605f704-serviceca\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837297    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-netns\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837321    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837346    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kmt\" (UniqueName: \"kubernetes.io/projected/8222297a-ec44-495a-b6bd-6b58501aadd4-kube-api-access-44kmt\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837389    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-data-volume\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837413    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-ovn\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837441    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-socket-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837467    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-data-volume\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837503    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kb4\" (UniqueName: \"kubernetes.io/projected/1e373a48-c67d-4cd3-8837-166a41802085-kube-api-access-p2kb4\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837526    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-kubernetes\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837543    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837568    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9699d94f-597e-4612-a469-3e657bfc9a55-signing-cabundle\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837628    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-agent-certs\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837646    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3138bcb-bcf2-49bc-959d-c2177605f704-serviceca\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837661    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-cnibin\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.837696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837689    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837737    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837760    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3138bcb-bcf2-49bc-959d-c2177605f704-host\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837785    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-dbus\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837800    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837828    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-log-socket\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837859    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-registration-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837908    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-agent-certs\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837910    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-conf-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837952    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-system-cni-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837964    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-original-pull-secret\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.837982    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-crio-socket\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838003    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838033    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnh6\" (UniqueName: \"kubernetes.io/projected/2c073539-dddd-4fd2-88ee-52fc61b9f50d-kube-api-access-2pnh6\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838115    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysconfig\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838141    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42f2ccd-9e73-4f22-8370-409ca511aeb3-config-volume\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838165    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-konnectivity-ca\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838199    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-config\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838233    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-env-overrides\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838249    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838265    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-sys-fs\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838280    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-run\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838296    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-host\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838341    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-tmp\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838345    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1a0388c-1177-45a8-8eb8-92ef23d8f0bf-konnectivity-ca\") pod \"konnectivity-agent-t4sns\" (UID: \"c1a0388c-1177-45a8-8eb8-92ef23d8f0bf\") " pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838370    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-env-overrides\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838389    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-kubelet-config\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838423    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-tmp\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838432    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-ovnkube-config\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838474    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-systemd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838501    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-netd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838531    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838555    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-systemd\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838583    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstsl\" (UniqueName: \"kubernetes.io/projected/c42f2ccd-9e73-4f22-8370-409ca511aeb3-kube-api-access-pstsl\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838609    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-netns\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838633    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838637    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838672    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.838941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.838840    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8222297a-ec44-495a-b6bd-6b58501aadd4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.843092 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.843078    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:33:41.843141 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.843117    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.845087 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.845074    4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 17:33:41.845136 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.845103    4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-172.us-west-1.compute.internal"
Nov 03 17:33:41.891694 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.891636    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 17:33:41.891777 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.891754    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.891845 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.891771    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.892073 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892052    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.892138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892076    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.892138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892076    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.892138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892093    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 17:33:41.892292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892143    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.892292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892174    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.892292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892226    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.892441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892298    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 17:33:41.892441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892322    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 17:33:41.892441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892391    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.892830 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892816    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 17:33:41.892830 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892819    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.892971 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892852    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 17:33:41.892971 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892899    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 17:33:41.893121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892974    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 17:33:41.893121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.892976    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-htgnv\""
Nov 03 17:33:41.893121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.893012    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 17:33:41.893121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.893082    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tz4q7\""
Nov 03 17:33:41.893121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.893107    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2pwfx\""
Nov 03 17:33:41.893285 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.893150    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jj5j\""
Nov 03 17:33:41.893285 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.893199    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2vfjt\""
Nov 03 17:33:41.893285 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.893212    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-df4kg\""
Nov 03 17:33:41.893542 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.893530    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 17:33:41.896820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.896806    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.896967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.896953    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9699d94f-597e-4612-a469-3e657bfc9a55-signing-key\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:41.897709 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.897693    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56t79\" (UniqueName: \"kubernetes.io/projected/60cede5f-b376-45dd-b774-feb3598cf479-kube-api-access-56t79\") pod \"migrator-66df5d496b-xjvqh\" (UID: \"60cede5f-b376-45dd-b774-feb3598cf479\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh"
Nov 03 17:33:41.897866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.897850    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9699d94f-597e-4612-a469-3e657bfc9a55-signing-cabundle\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:41.898075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898056    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bstfb\""
Nov 03 17:33:41.898227 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898210    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnnd\" (UniqueName: \"kubernetes.io/projected/1758fefb-559e-433d-a077-bb4b574853e1-kube-api-access-lbnnd\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:41.898296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898228    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdj6s\" (UniqueName: \"kubernetes.io/projected/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-kube-api-access-sdj6s\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:41.898379 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898357    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vjp\" (UniqueName: \"kubernetes.io/projected/d3138bcb-bcf2-49bc-959d-c2177605f704-kube-api-access-g4vjp\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:41.898449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898430    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c42f2ccd-9e73-4f22-8370-409ca511aeb3-config-volume\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.898717 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898699    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5l8w\" (UniqueName: \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.898717 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898710    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458s7\" (UniqueName: \"kubernetes.io/projected/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-api-access-458s7\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.898789 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898704    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2th75\" (UniqueName: \"kubernetes.io/projected/01b8876a-2097-40ee-9b09-4ec46606a7b1-kube-api-access-2th75\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.898938 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.898923    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9pr\" (UniqueName: \"kubernetes.io/projected/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-kube-api-access-pz9pr\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.899588 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.899570    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtkn\" (UniqueName: \"kubernetes.io/projected/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-kube-api-access-lbtkn\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:41.899828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.899808    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\") pod \"periodic-gathering-fssvd-2gvm4\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") " pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:33:41.899898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.899863    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm67z\" (UniqueName: \"kubernetes.io/projected/9699d94f-597e-4612-a469-3e657bfc9a55-kube-api-access-nm67z\") pod \"service-ca-5fffc97bdf-gjgn4\" (UID: \"9699d94f-597e-4612-a469-3e657bfc9a55\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4"
Nov 03 17:33:41.900730 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.900718    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 17:33:41.903599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.903584    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kmt\" (UniqueName: \"kubernetes.io/projected/8222297a-ec44-495a-b6bd-6b58501aadd4-kube-api-access-44kmt\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.904253 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.904236    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstsl\" (UniqueName: \"kubernetes.io/projected/c42f2ccd-9e73-4f22-8370-409ca511aeb3-kube-api-access-pstsl\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.904830 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.904799    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnh6\" (UniqueName: \"kubernetes.io/projected/2c073539-dddd-4fd2-88ee-52fc61b9f50d-kube-api-access-2pnh6\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:41.905049 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.905035    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km7g\" (UniqueName: \"kubernetes.io/projected/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-kube-api-access-5km7g\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.905138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.905123    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5czc\" (UniqueName: \"kubernetes.io/projected/44f3aa72-92ab-4dff-9c6a-2c78626e8b50-kube-api-access-v5czc\") pod \"network-check-target-7v8km\" (UID: \"44f3aa72-92ab-4dff-9c6a-2c78626e8b50\") " pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:41.906308 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.906292    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kb4\" (UniqueName: \"kubernetes.io/projected/1e373a48-c67d-4cd3-8837-166a41802085-kube-api-access-p2kb4\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.909679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.909665    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 17:33:41.915651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.915635    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1758fefb-559e-433d-a077-bb4b574853e1-cert\") pod \"ingress-canary-6shfz\" (UID: \"1758fefb-559e-433d-a077-bb4b574853e1\") " pod="openshift-ingress-canary/ingress-canary-6shfz"
Nov 03 17:33:41.939320 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939294    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-lib-modules\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.939415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939324    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-hostroot\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.939415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939350    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-var-lib-kubelet\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.939415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939373    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-bin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.939415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939397    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-netns\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.939415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939400    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-hostroot\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939421    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939428    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-var-lib-kubelet\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939437    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-lib-modules\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939438    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-bin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939449    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-ovn\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939459    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939475    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-netns\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939503    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-socket-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939529    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-ovn\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939545    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-kubernetes\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939612    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-socket-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.939626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939616    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939643    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-kubernetes\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939685    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-cnibin\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939706    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.939716    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939734    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-cnibin\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939772    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3138bcb-bcf2-49bc-959d-c2177605f704-host\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939803    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3138bcb-bcf2-49bc-959d-c2177605f704-host\") pod \"node-ca-9n9mf\" (UID: \"d3138bcb-bcf2-49bc-959d-c2177605f704\") " pod="openshift-image-registry/node-ca-9n9mf"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939831    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-dbus\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939859    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939907    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-log-socket\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939918    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-dbus\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939932    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-registration-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.939938    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.939952    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.439938057 +0000 UTC m=+1.899109037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939961    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.939982    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.439967152 +0000 UTC m=+1.899138118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : secret "metrics-daemon-secret" not found
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939982    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-registration-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.939990    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-log-socket\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940000    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-conf-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940019    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-system-cni-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940035    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-crio-socket\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940043    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-system-cni-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940022    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-conf-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940057    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysconfig\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940085    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-crio-socket\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.940080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940086    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940097    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysconfig\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940111    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-sys-fs\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940127    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-run\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940124    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940143    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-host\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940162    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-sys-fs\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940182    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-run\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940184    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-kubelet-config\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940205    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-systemd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940211    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b82e04a-d3d2-43de-ad90-abccdeb48e7e-kubelet-config\") pod \"global-pull-secret-syncer-648bg\" (UID: \"9b82e04a-d3d2-43de-ad90-abccdeb48e7e\") " pod="kube-system/global-pull-secret-syncer-648bg"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940231    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-netd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940233    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-host\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940249    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-systemd\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940262    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-run-systemd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940298    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-netd\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940323    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-netns\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940345    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-systemd\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940372    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940387    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-var-lib-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940389    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-netns\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940404    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940415    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-var-lib-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940420    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-multus-certs\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940430    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940438    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-multus\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940455    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-kubelet\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940469    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-systemd-units\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940473    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-cni-multus\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940483    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-conf\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940456    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-multus-certs\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940567    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-var-lib-kubelet\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.940476    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940574    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-systemd-units\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940593    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-conf\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940613    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-sys\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940682    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-host-slash\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940683    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-sys\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.940743    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.440733484 +0000 UTC m=+1.899904448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940741    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef771bc1-ab90-44d0-93a5-5e3fa2a8db40-host-slash\") pod \"iptables-alerter-88fkm\" (UID: \"ef771bc1-ab90-44d0-93a5-5e3fa2a8db40\") " pod="openshift-network-operator/iptables-alerter-88fkm"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940759    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-etc-kubernetes\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940794    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-kubelet\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940808    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-etc-kubernetes\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940808    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940832    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-system-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940839    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940847    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-k8s-cni-cncf-io\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940862    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-kubelet\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940863    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940911    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-system-cni-dir\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940909    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8222297a-ec44-495a-b6bd-6b58501aadd4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vfd7n\" (UID: \"8222297a-ec44-495a-b6bd-6b58501aadd4\") " pod="openshift-multus/multus-additional-cni-plugins-vfd7n"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940934    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940937    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-host-run-k8s-cni-cncf-io\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.940922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940957    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-device-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940971    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-modprobe-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940984    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.940997    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-cnibin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941025    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1e373a48-c67d-4cd3-8837-166a41802085-device-dir\") pod \"aws-ebs-csi-driver-node-z8hrl\" (UID: \"1e373a48-c67d-4cd3-8837-166a41802085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8hrl"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941030    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-socket-dir-parent\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.941059    4493 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941076    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-slash\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941088    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-cnibin\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941086    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-modprobe-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941105    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-etc-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941112    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5cc09b0-ed27-4664-9d61-81d7e3f7ef09-etc-sysctl-d\") pod \"tuned-vcxvs\" (UID: \"c5cc09b0-ed27-4664-9d61-81d7e3f7ef09\") " pod="openshift-cluster-node-tuning-operator/tuned-vcxvs"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:41.941122    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.441107934 +0000 UTC m=+1.900278918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : configmap "kube-rbac-proxy" not found
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941119    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/01b8876a-2097-40ee-9b09-4ec46606a7b1-multus-socket-dir-parent\") pod \"multus-7qrt7\" (UID: \"01b8876a-2097-40ee-9b09-4ec46606a7b1\") " pod="openshift-multus/multus-7qrt7"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941140    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-etc-openvswitch\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941142    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-slash\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941158    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-node-log\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941188    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941200    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-node-log\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941217    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-bin\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941233    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941249    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c073539-dddd-4fd2-88ee-52fc61b9f50d-hosts-file\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941251    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c37bc06-03be-48f6-9bd9-f6eb3e711ef0-host-cni-bin\") pod \"ovnkube-node-rgmgf\" (UID: \"6c37bc06-03be-48f6-9bd9-f6eb3e711ef0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:41.942411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:41.941298    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c073539-dddd-4fd2-88ee-52fc61b9f50d-hosts-file\") pod \"node-resolver-hcf4k\" (UID: \"2c073539-dddd-4fd2-88ee-52fc61b9f50d\") " pod="openshift-dns/node-resolver-hcf4k"
Nov 03 17:33:42.031927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:42.031906    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:33:42.444457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:42.444416    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:42.444457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:42.444461    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:42.444715 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444538    4493 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 17:33:42.444715 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444583    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:42.444715 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:42.444592    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:42.444715 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:42.444634    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:42.444715 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444646    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:42.444715 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444650    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.444631847 +0000 UTC m=+2.903802816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:42.444715 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444709    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.444697426 +0000 UTC m=+2.903868389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : configmap "kube-rbac-proxy" not found
Nov 03 17:33:42.444999 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444725    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.444715555 +0000 UTC m=+2.903886524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:42.444999 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444729    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 17:33:42.444999 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:42.444782    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.444768815 +0000 UTC m=+2.903939798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : secret "metrics-daemon-secret" not found
Nov 03 17:33:42.613553 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:42.613521    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:43.453556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:43.453475    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:43.453556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:43.453539    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:43.453568    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:43.453607    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:43.453653    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:43.453704    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:43.453706    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:43.453713    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.453696202 +0000 UTC m=+4.912867166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : secret "metrics-daemon-secret" not found
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:43.453772    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.453756775 +0000 UTC m=+4.912927739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:43.454031 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:43.453788    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.453778083 +0000 UTC m=+4.912949052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:43.454390 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:43.454097    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:43.758907 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:43.758841    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xjvqh" event={"ID":"60cede5f-b376-45dd-b774-feb3598cf479","Type":"ContainerStarted","Data":"68e29703701a244628ba57966a4b4bf19655cca13082d044c0007979e2c14fe5"}
Nov 03 17:33:43.760563 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:43.760529    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fssvd-2gvm4" event={"ID":"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9","Type":"ContainerStarted","Data":"4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee"}
Nov 03 17:33:45.472030 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:45.471979    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:45.472030 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:45.472033    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:45.472567 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:45.472084    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:45.472567 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:45.472149    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:45.472567 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:45.472160    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:45.472567 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:45.472219    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.472204212 +0000 UTC m=+8.931375196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:45.472567 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:45.472231    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.472226414 +0000 UTC m=+8.931397381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:45.472567 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:45.472245    4493 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 17:33:45.472567 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:45.472320    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs podName:c85b087b-5e24-45d0-abe3-d8e72dbbb74e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.472303279 +0000 UTC m=+8.931474261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs") pod "network-metrics-daemon-2ccdq" (UID: "c85b087b-5e24-45d0-abe3-d8e72dbbb74e") : secret "metrics-daemon-secret" not found
Nov 03 17:33:45.769076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:45.768984    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-gjgn4" event={"ID":"9699d94f-597e-4612-a469-3e657bfc9a55","Type":"ContainerStarted","Data":"8218ba959c55e770a52a1a52258ffa8c100054181cf7c160f56f54b707244d83"}
Nov 03 17:33:47.269921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:47.269895    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:33:47.270390 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:47.270136    4493 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 17:33:48.051693 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:48.051657    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:33:48.755930 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:48.755905    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hcf4k_2c073539-dddd-4fd2-88ee-52fc61b9f50d/dns-node-resolver/0.log"
Nov 03 17:33:49.355925 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.355900    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9n9mf_d3138bcb-bcf2-49bc-959d-c2177605f704/node-ca/0.log"
Nov 03 17:33:49.504437 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.504402    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:49.504588 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.504453    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:49.504588 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:49.504540    4493 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 17:33:49.504588 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:49.504545    4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:49.504588 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.504573    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:49.504715 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:49.504594    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls podName:c42f2ccd-9e73-4f22-8370-409ca511aeb3 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.504582188 +0000 UTC m=+16.963753153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls") pod "dns-default-4w9qr" (UID: "c42f2ccd-9e73-4f22-8370-409ca511aeb3") : secret "dns-default-metrics-tls" not found
Nov 03 17:33:49.504715 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:33:49.504622    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls podName:944b1b52-5928-4c8d-9d19-f6a17c45f9e7 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.504612699 +0000 UTC m=+16.963783664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-82x2r" (UID: "944b1b52-5928-4c8d-9d19-f6a17c45f9e7") : secret "insights-runtime-extractor-tls" not found
Nov 03 17:33:49.504922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.504905    4493 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 17:33:49.507109 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.507089    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85b087b-5e24-45d0-abe3-d8e72dbbb74e-metrics-certs\") pod \"network-metrics-daemon-2ccdq\" (UID: \"c85b087b-5e24-45d0-abe3-d8e72dbbb74e\") " pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:49.527161 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.527132    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2ccdq"
Nov 03 17:33:49.894939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:49.894908    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2ccdq"]
Nov 03 17:33:49.898053 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:49.898023    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85b087b_5e24_45d0_abe3_d8e72dbbb74e.slice/crio-2905008343c111d31009a82284078adb84cd6a1666ac71bb52fd644e000654b5 WatchSource:0}: Error finding container 2905008343c111d31009a82284078adb84cd6a1666ac71bb52fd644e000654b5: Status 404 returned error can't find the container with id 2905008343c111d31009a82284078adb84cd6a1666ac71bb52fd644e000654b5
Nov 03 17:33:50.156651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:50.156568    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6shfz_1758fefb-559e-433d-a077-bb4b574853e1/serve-healthcheck-canary/0.log"
Nov 03 17:33:50.757134 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:50.757110    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xjvqh_60cede5f-b376-45dd-b774-feb3598cf479/migrator/0.log"
Nov 03 17:33:50.780780 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:50.780746    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ccdq" event={"ID":"c85b087b-5e24-45d0-abe3-d8e72dbbb74e","Type":"ContainerStarted","Data":"2905008343c111d31009a82284078adb84cd6a1666ac71bb52fd644e000654b5"}
Nov 03 17:33:51.260502 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:51.260469    4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-172.us-west-1.compute.internal" event="NodeReady"
Nov 03 17:33:51.784857 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:51.784782    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ccdq" event={"ID":"c85b087b-5e24-45d0-abe3-d8e72dbbb74e","Type":"ContainerStarted","Data":"77c0c1a71e0db6db7e4b60e2a58bcf5439792f01c312ff45357ad4dd1be58333"}
Nov 03 17:33:51.784857 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:51.784834    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2ccdq" event={"ID":"c85b087b-5e24-45d0-abe3-d8e72dbbb74e","Type":"ContainerStarted","Data":"78e688b692a613b5ef0f1882a74953618990f55686eee6c7cce44e08792b41ff"}
Nov 03 17:33:57.561450 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:57.561410    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:57.561915 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:57.561466    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:57.564008 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:57.563980    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/944b1b52-5928-4c8d-9d19-f6a17c45f9e7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-82x2r\" (UID: \"944b1b52-5928-4c8d-9d19-f6a17c45f9e7\") " pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:57.564121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:57.563984    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c42f2ccd-9e73-4f22-8370-409ca511aeb3-metrics-tls\") pod \"dns-default-4w9qr\" (UID: \"c42f2ccd-9e73-4f22-8370-409ca511aeb3\") " pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:57.630209 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:57.630176    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:33:57.631309 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:57.631285    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-82x2r"
Nov 03 17:33:58.012240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:58.012203    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4w9qr"]
Nov 03 17:33:58.015378 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:58.015349    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42f2ccd_9e73_4f22_8370_409ca511aeb3.slice/crio-4ef557c07b05caad358f347eb7f7a8b1acedee61487e717075f4a3fc45ba9675 WatchSource:0}: Error finding container 4ef557c07b05caad358f347eb7f7a8b1acedee61487e717075f4a3fc45ba9675: Status 404 returned error can't find the container with id 4ef557c07b05caad358f347eb7f7a8b1acedee61487e717075f4a3fc45ba9675
Nov 03 17:33:58.062895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:58.062847    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-82x2r"]
Nov 03 17:33:58.065834 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:33:58.065810    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944b1b52_5928_4c8d_9d19_f6a17c45f9e7.slice/crio-7b4115a2a67426d1d8cb8178b84d64436014ecc42602ed30f751243fff417506 WatchSource:0}: Error finding container 7b4115a2a67426d1d8cb8178b84d64436014ecc42602ed30f751243fff417506: Status 404 returned error can't find the container with id 7b4115a2a67426d1d8cb8178b84d64436014ecc42602ed30f751243fff417506
Nov 03 17:33:58.808308 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:58.808277    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82x2r" event={"ID":"944b1b52-5928-4c8d-9d19-f6a17c45f9e7","Type":"ContainerStarted","Data":"42c26c9765bfccfe655e8215f060effaf9c4c3ca94e15c7971eb6a423f8d5468"}
Nov 03 17:33:58.808751 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:58.808338    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82x2r" event={"ID":"944b1b52-5928-4c8d-9d19-f6a17c45f9e7","Type":"ContainerStarted","Data":"7b4115a2a67426d1d8cb8178b84d64436014ecc42602ed30f751243fff417506"}
Nov 03 17:33:58.809508 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:58.809483    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4w9qr" event={"ID":"c42f2ccd-9e73-4f22-8370-409ca511aeb3","Type":"ContainerStarted","Data":"4ef557c07b05caad358f347eb7f7a8b1acedee61487e717075f4a3fc45ba9675"}
Nov 03 17:33:59.814649 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:33:59.814612    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82x2r" event={"ID":"944b1b52-5928-4c8d-9d19-f6a17c45f9e7","Type":"ContainerStarted","Data":"525b4073ab189041f3b34c84693b2db47f8d609865bcc82420c5287aeafc8dda"}
Nov 03 17:34:00.823894 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:00.823817    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4w9qr" event={"ID":"c42f2ccd-9e73-4f22-8370-409ca511aeb3","Type":"ContainerStarted","Data":"89dab7cfb8316fdab959ba3a1d76856b5c583a7d80864229e22a8ccbf2b2b80f"}
Nov 03 17:34:00.824285 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:00.823958    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4w9qr" event={"ID":"c42f2ccd-9e73-4f22-8370-409ca511aeb3","Type":"ContainerStarted","Data":"bd93459e0b662ede5175c16a32700d23f10beab3d4dcfcc889655b840680cfa9"}
Nov 03 17:34:00.824351 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:00.824300    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:34:01.182142 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.182104    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-59cc2"]
Nov 03 17:34:01.184124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.184107    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.238867 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.238830    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 17:34:01.239124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.238940    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 17:34:01.261453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.261420    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-59cc2"]
Nov 03 17:34:01.287044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.287010    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkm7\" (UniqueName: \"kubernetes.io/projected/5b8cb8e3-c28a-4873-821b-8fae7a588144-kube-api-access-xgkm7\") pod \"downloads-6dd789c8b7-59cc2\" (UID: \"5b8cb8e3-c28a-4873-821b-8fae7a588144\") " pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.287212 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.287086    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8cb8e3-c28a-4873-821b-8fae7a588144-tmp\") pod \"downloads-6dd789c8b7-59cc2\" (UID: \"5b8cb8e3-c28a-4873-821b-8fae7a588144\") " pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.293377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.293342    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86c49c9b68-2pxdq"]
Nov 03 17:34:01.295137 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.295118    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.323084 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.323055    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-x8zsr\""
Nov 03 17:34:01.388433 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.388348    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8cb8e3-c28a-4873-821b-8fae7a588144-tmp\") pod \"downloads-6dd789c8b7-59cc2\" (UID: \"5b8cb8e3-c28a-4873-821b-8fae7a588144\") " pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.388433 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.388392    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkm7\" (UniqueName: \"kubernetes.io/projected/5b8cb8e3-c28a-4873-821b-8fae7a588144-kube-api-access-xgkm7\") pod \"downloads-6dd789c8b7-59cc2\" (UID: \"5b8cb8e3-c28a-4873-821b-8fae7a588144\") " pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.388824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.388800    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8cb8e3-c28a-4873-821b-8fae7a588144-tmp\") pod \"downloads-6dd789c8b7-59cc2\" (UID: \"5b8cb8e3-c28a-4873-821b-8fae7a588144\") " pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.400299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.400257    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 17:34:01.400449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.400297    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 17:34:01.400449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.400327    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 17:34:01.401043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.401023    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86c49c9b68-2pxdq"]
Nov 03 17:34:01.408035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.408015    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lr55j\""
Nov 03 17:34:01.408035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.408010    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 17:34:01.452164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.452131    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkm7\" (UniqueName: \"kubernetes.io/projected/5b8cb8e3-c28a-4873-821b-8fae7a588144-kube-api-access-xgkm7\") pod \"downloads-6dd789c8b7-59cc2\" (UID: \"5b8cb8e3-c28a-4873-821b-8fae7a588144\") " pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.489097 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489059    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f6d08ad-7488-49a3-9281-eff56a048b5b-trusted-ca\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.489280 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489105    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f6d08ad-7488-49a3-9281-eff56a048b5b-installation-pull-secrets\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.489280 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489128    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f6d08ad-7488-49a3-9281-eff56a048b5b-image-registry-private-configuration\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.489280 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489246    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f6d08ad-7488-49a3-9281-eff56a048b5b-registry-certificates\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.489406 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489292    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkz69\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-kube-api-access-tkz69\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.489406 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489315    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-registry-tls\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.489406 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489337    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-bound-sa-token\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.489406 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.489359    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f6d08ad-7488-49a3-9281-eff56a048b5b-ca-trust-extracted\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.494128 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.494108    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:01.590575 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590520    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f6d08ad-7488-49a3-9281-eff56a048b5b-trusted-ca\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.590575 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590581    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f6d08ad-7488-49a3-9281-eff56a048b5b-installation-pull-secrets\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.590792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590602    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f6d08ad-7488-49a3-9281-eff56a048b5b-image-registry-private-configuration\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.590792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590621    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f6d08ad-7488-49a3-9281-eff56a048b5b-registry-certificates\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.590792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590643    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkz69\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-kube-api-access-tkz69\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.590792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590661    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-registry-tls\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.590792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590701    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-bound-sa-token\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.590792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.590738    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f6d08ad-7488-49a3-9281-eff56a048b5b-ca-trust-extracted\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.591572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.591444    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f6d08ad-7488-49a3-9281-eff56a048b5b-ca-trust-extracted\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.591845 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.591810    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f6d08ad-7488-49a3-9281-eff56a048b5b-registry-certificates\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.592193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.592174    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f6d08ad-7488-49a3-9281-eff56a048b5b-trusted-ca\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.593454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.593434    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f6d08ad-7488-49a3-9281-eff56a048b5b-installation-pull-secrets\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.593529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.593434    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f6d08ad-7488-49a3-9281-eff56a048b5b-image-registry-private-configuration\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.593573 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.593554    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-registry-tls\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.654889 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.654806    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkz69\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-kube-api-access-tkz69\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.655021 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.654938    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f6d08ad-7488-49a3-9281-eff56a048b5b-bound-sa-token\") pod \"image-registry-86c49c9b68-2pxdq\" (UID: \"9f6d08ad-7488-49a3-9281-eff56a048b5b\") " pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.829095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.829056    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-82x2r" event={"ID":"944b1b52-5928-4c8d-9d19-f6a17c45f9e7","Type":"ContainerStarted","Data":"9211fe9d8092d20e1a3a4d8d61bd72a7e5a07c0b9fcc605d5748097e7f357503"}
Nov 03 17:34:01.904519 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.904480    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:01.918338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:01.918312    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-59cc2"]
Nov 03 17:34:01.922101 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:34:01.922075    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8cb8e3_c28a_4873_821b_8fae7a588144.slice/crio-4a3b6d3ba3e687c5461149b1368ff7e917d5ff1c72853e71ff9d042e6a0670d4 WatchSource:0}: Error finding container 4a3b6d3ba3e687c5461149b1368ff7e917d5ff1c72853e71ff9d042e6a0670d4: Status 404 returned error can't find the container with id 4a3b6d3ba3e687c5461149b1368ff7e917d5ff1c72853e71ff9d042e6a0670d4
Nov 03 17:34:02.283830 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:02.283800    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86c49c9b68-2pxdq"]
Nov 03 17:34:02.286781 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:34:02.286756    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6d08ad_7488_49a3_9281_eff56a048b5b.slice/crio-09c928eccde2b32a5bee3eff1400eb353910cb9c94deb249317464417c30da95 WatchSource:0}: Error finding container 09c928eccde2b32a5bee3eff1400eb353910cb9c94deb249317464417c30da95: Status 404 returned error can't find the container with id 09c928eccde2b32a5bee3eff1400eb353910cb9c94deb249317464417c30da95
Nov 03 17:34:02.835212 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:02.835165    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-59cc2" event={"ID":"5b8cb8e3-c28a-4873-821b-8fae7a588144","Type":"ContainerStarted","Data":"4a3b6d3ba3e687c5461149b1368ff7e917d5ff1c72853e71ff9d042e6a0670d4"}
Nov 03 17:34:02.837471 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:02.836960    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq" event={"ID":"9f6d08ad-7488-49a3-9281-eff56a048b5b","Type":"ContainerStarted","Data":"82e6f5c65ccbd97fc9e6062a533a845d5f3ebb21905d3005aae5197a288d16d0"}
Nov 03 17:34:02.837471 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:02.837011    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq" event={"ID":"9f6d08ad-7488-49a3-9281-eff56a048b5b","Type":"ContainerStarted","Data":"09c928eccde2b32a5bee3eff1400eb353910cb9c94deb249317464417c30da95"}
Nov 03 17:34:02.837471 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:02.837195    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:02.964014 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:02.963944    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq" podStartSLOduration=1.963926361 podStartE2EDuration="1.963926361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:02.962010979 +0000 UTC m=+22.421181957" watchObservedRunningTime="2026-11-03 17:34:02.963926361 +0000 UTC m=+22.423097346"
Nov 03 17:34:05.527043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:05.526001    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rgmgf"
Nov 03 17:34:06.453609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:06.453578    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t4sns"
Nov 03 17:34:06.995513 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:34:06.995478    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:34:08.055017 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:08.054983    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7v8km"
Nov 03 17:34:10.831412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:10.831373    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4w9qr"
Nov 03 17:34:14.126264 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.126235    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qss6d"]
Nov 03 17:34:14.163579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.163539    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.175994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.175963    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-tls\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176137 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176012    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176137 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176096    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-sys\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176138    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8188d-5925-4eca-af95-23c3efd5a999-metrics-client-ca\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176173    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-wtmp\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176294    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-accelerators-collector-config\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176335    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-textfile\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176435 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176361    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdtx\" (UniqueName: \"kubernetes.io/projected/b5b8188d-5925-4eca-af95-23c3efd5a999-kube-api-access-hgdtx\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.176478 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.176454    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-root\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.262927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.262893    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 17:34:14.272063 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.272025    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8sjq\""
Nov 03 17:34:14.272063 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.272055    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 17:34:14.272334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.272027    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 17:34:14.272334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.272114    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 17:34:14.272334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.272178    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 17:34:14.272334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.272214    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 17:34:14.277673 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.277646    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-textfile\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.277803 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.277694    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdtx\" (UniqueName: \"kubernetes.io/projected/b5b8188d-5925-4eca-af95-23c3efd5a999-kube-api-access-hgdtx\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.277865 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.277820    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-root\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.277947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.277909    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-tls\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.277947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.277928    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-root\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.277947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.277940    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278001    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-sys\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278040    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8188d-5925-4eca-af95-23c3efd5a999-metrics-client-ca\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278047    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-textfile\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278088    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-wtmp\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278283 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278115    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-sys\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278283 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278159    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-accelerators-collector-config\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278283 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278238    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-wtmp\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278571 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278550    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8188d-5925-4eca-af95-23c3efd5a999-metrics-client-ca\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.278743 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.278692    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-accelerators-collector-config\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.280985 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.280962    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.282237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.282218    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b8188d-5925-4eca-af95-23c3efd5a999-node-exporter-tls\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.342726 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.342682    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdtx\" (UniqueName: \"kubernetes.io/projected/b5b8188d-5925-4eca-af95-23c3efd5a999-kube-api-access-hgdtx\") pod \"node-exporter-qss6d\" (UID: \"b5b8188d-5925-4eca-af95-23c3efd5a999\") " pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.515759 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.515726    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qss6d"
Nov 03 17:34:14.547233 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:34:14.547201    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b8188d_5925_4eca_af95_23c3efd5a999.slice/crio-ef0ffc93c1bafa328aa85a161c18fe2349f1e9285a579b607c1d6d402bbc072d WatchSource:0}: Error finding container ef0ffc93c1bafa328aa85a161c18fe2349f1e9285a579b607c1d6d402bbc072d: Status 404 returned error can't find the container with id ef0ffc93c1bafa328aa85a161c18fe2349f1e9285a579b607c1d6d402bbc072d
Nov 03 17:34:14.880172 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:14.880082    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss6d" event={"ID":"b5b8188d-5925-4eca-af95-23c3efd5a999","Type":"ContainerStarted","Data":"ef0ffc93c1bafa328aa85a161c18fe2349f1e9285a579b607c1d6d402bbc072d"}
Nov 03 17:34:16.889627 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:16.889543    4493 generic.go:356] "Generic (PLEG): container finished" podID="b5b8188d-5925-4eca-af95-23c3efd5a999" containerID="0a65dffc6647e1dce51af2e30edb21fcc75ce7ab7db1ab415645d45b84d95e80" exitCode=0
Nov 03 17:34:16.890068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:16.889637    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss6d" event={"ID":"b5b8188d-5925-4eca-af95-23c3efd5a999","Type":"ContainerDied","Data":"0a65dffc6647e1dce51af2e30edb21fcc75ce7ab7db1ab415645d45b84d95e80"}
Nov 03 17:34:17.894609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:17.894576    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-59cc2" event={"ID":"5b8cb8e3-c28a-4873-821b-8fae7a588144","Type":"ContainerStarted","Data":"ae456dc3194970f43884e30532cf575d4c49a4b1d88298b072ce085ef73688d0"}
Nov 03 17:34:17.895095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:17.894769    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:17.896594 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:17.896561    4493 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-59cc2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.12:8080/\": dial tcp 10.133.0.12:8080: connect: connection refused" start-of-body=
Nov 03 17:34:17.896707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:17.896606    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-59cc2" podUID="5b8cb8e3-c28a-4873-821b-8fae7a588144" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.12:8080/\": dial tcp 10.133.0.12:8080: connect: connection refused"
Nov 03 17:34:17.897402 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:17.897364    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss6d" event={"ID":"b5b8188d-5925-4eca-af95-23c3efd5a999","Type":"ContainerStarted","Data":"dba3f9d41c8cc585e65c8ce495b34959ced059df6eba16b593a29368b43a2b7d"}
Nov 03 17:34:17.897488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:17.897418    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss6d" event={"ID":"b5b8188d-5925-4eca-af95-23c3efd5a999","Type":"ContainerStarted","Data":"5f3afb5cab65dc09bc917f4b10d7ac8cf7e0ed33e080f20c0d91e117bd75b385"}
Nov 03 17:34:18.021155 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:18.021044    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-59cc2" podStartSLOduration=1.189010897 podStartE2EDuration="17.02103045s" totalImagesPullingTime="15.832019553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:18.020067133 +0000 UTC m=+37.479238140" watchObservedRunningTime="2026-11-03 17:34:18.02103045 +0000 UTC m=+37.480201436"
Nov 03 17:34:18.147487 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:18.147400    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qss6d" podStartSLOduration=2.109227096 podStartE2EDuration="4.147380414s" totalImagesPullingTime="1.937929363s" totalInitContainerRuntime="100.223955ms" isStatefulPod=true podCreationTimestamp="2026-11-03 17:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:18.146690267 +0000 UTC m=+37.605861263" watchObservedRunningTime="2026-11-03 17:34:18.147380414 +0000 UTC m=+37.606551401"
Nov 03 17:34:18.930335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:18.930298    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-59cc2"
Nov 03 17:34:19.110255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.110217    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64477db88b-vv6ll"]
Nov 03 17:34:19.112440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.112417    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.167882 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.167840    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 17:34:19.168074 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.167841    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 17:34:19.168204 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.168162    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 17:34:19.168368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.168347    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qr8c5\""
Nov 03 17:34:19.168517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.168497    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 17:34:19.168517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.168392    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b7tugtncran1k\""
Nov 03 17:34:19.174854 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.174824    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64477db88b-vv6ll"]
Nov 03 17:34:19.223037 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.223001    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d82d34-457f-4bbd-b196-9047c48ee579-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.223037 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.223050    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-client-ca-bundle\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.223278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.223126    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92w75\" (UniqueName: \"kubernetes.io/projected/a9d82d34-457f-4bbd-b196-9047c48ee579-kube-api-access-92w75\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.223278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.223157    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9d82d34-457f-4bbd-b196-9047c48ee579-audit-log\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.223278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.223181    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-secret-metrics-server-tls\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.223446 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.223334    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-secret-metrics-server-client-certs\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.223446 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.223401    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9d82d34-457f-4bbd-b196-9047c48ee579-metrics-server-audit-profiles\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.324827 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.324784    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-secret-metrics-server-client-certs\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.325034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.324836    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9d82d34-457f-4bbd-b196-9047c48ee579-metrics-server-audit-profiles\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.325034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.324906    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d82d34-457f-4bbd-b196-9047c48ee579-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.325034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.324933    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-client-ca-bundle\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.325034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.324969    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92w75\" (UniqueName: \"kubernetes.io/projected/a9d82d34-457f-4bbd-b196-9047c48ee579-kube-api-access-92w75\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.325034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.325004    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9d82d34-457f-4bbd-b196-9047c48ee579-audit-log\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.325288 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.325035    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-secret-metrics-server-tls\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.325739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.325683    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a9d82d34-457f-4bbd-b196-9047c48ee579-audit-log\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.326083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.326051    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a9d82d34-457f-4bbd-b196-9047c48ee579-metrics-server-audit-profiles\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.326474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.326449    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d82d34-457f-4bbd-b196-9047c48ee579-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.328132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.328111    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-client-ca-bundle\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.328267 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.328165    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-secret-metrics-server-tls\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.328348 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.328327    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d82d34-457f-4bbd-b196-9047c48ee579-secret-metrics-server-client-certs\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.390714 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.390678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92w75\" (UniqueName: \"kubernetes.io/projected/a9d82d34-457f-4bbd-b196-9047c48ee579-kube-api-access-92w75\") pod \"metrics-server-64477db88b-vv6ll\" (UID: \"a9d82d34-457f-4bbd-b196-9047c48ee579\") " pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.425599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.425561    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:19.867225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.867140    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64477db88b-vv6ll"]
Nov 03 17:34:19.870780 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:34:19.870746    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d82d34_457f_4bbd_b196_9047c48ee579.slice/crio-50e6941747a6b10f6d48ba64afb4d644661de6268881e286218775ce8aff3699 WatchSource:0}: Error finding container 50e6941747a6b10f6d48ba64afb4d644661de6268881e286218775ce8aff3699: Status 404 returned error can't find the container with id 50e6941747a6b10f6d48ba64afb4d644661de6268881e286218775ce8aff3699
Nov 03 17:34:19.909665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:19.909633    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll" event={"ID":"a9d82d34-457f-4bbd-b196-9047c48ee579","Type":"ContainerStarted","Data":"50e6941747a6b10f6d48ba64afb4d644661de6268881e286218775ce8aff3699"}
Nov 03 17:34:22.924389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:22.924304    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll" event={"ID":"a9d82d34-457f-4bbd-b196-9047c48ee579","Type":"ContainerStarted","Data":"f5eca6ee6eece32eec0841948bfbf99b76a1600d762a4e78fecabe8df89d76d9"}
Nov 03 17:34:23.050330 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.050278    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll" podStartSLOduration=1.4549313910000001 podStartE2EDuration="4.050264624s" totalImagesPullingTime="2.595333233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:23.049570291 +0000 UTC m=+42.508741298" watchObservedRunningTime="2026-11-03 17:34:23.050264624 +0000 UTC m=+42.509435661"
Nov 03 17:34:23.681959 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.681920    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:34:23.702233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.702196    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.757837 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.757804    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 17:34:23.757837 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.757830    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 17:34:23.766353 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766317    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766360    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766384    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766411    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766569    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8tm\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-kube-api-access-df8tm\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766604    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766622    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766726 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766707    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766755    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766833 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766812    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766846    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766904    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.766952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766942    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.767044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.766987    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.767044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.767028    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config-out\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.767116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.767080    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.767202 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.767187    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-web-config\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.767242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.767214    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.767290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.767244    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.767290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.767276    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.806958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.806914    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8f8m48m7s3jj2\""
Nov 03 17:34:23.806958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.806915    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 17:34:23.806958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.806912    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807020    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807032    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807062    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807114    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t8sg4\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807145    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807194    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807211    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807235    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 17:34:23.807401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.807365    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 17:34:23.846432 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.846401    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86c49c9b68-2pxdq"
Nov 03 17:34:23.868152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868127    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868156    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868175    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868196    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868249    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868284    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config-out\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868310    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868349    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-web-config\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868379    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868409    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868444    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868486    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868509    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868532    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868560    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868601    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8tm\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-kube-api-access-df8tm\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868638    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868661    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868693    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.868863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.868726    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.869696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.869669    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.871169 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.870335    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.873256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.872750    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.873256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.872831    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.873256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.871434    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.875414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.874280    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.875414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.874646    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.875414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.874704    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.875414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.874830    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.875414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.875101    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.875414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.875313    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.876024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.875931    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.876129 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.876076    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.876578 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.876550    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config-out\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.877509 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.877491    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.878409 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.878391    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-web-config\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.878503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.878418    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.908499 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.908471    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 17:34:23.909253 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.909227    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 17:34:23.910114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.910088    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:34:23.912843 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.912820    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.913781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.913759    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:23.939067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:23.938969    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8tm\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-kube-api-access-df8tm\") pod \"prometheus-k8s-0\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:24.165319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:24.165280    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:24.460927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:24.460805    4493 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t4sns_c1a0388c-1177-45a8-8eb8-92ef23d8f0bf/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:24.643291 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:24.643260    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:34:24.647156 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:34:24.647127    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb49ab7_ea8a_4d38_b284_98bfc251aafe.slice/crio-e282f4b8bfd50f1b751675292338850c5724784d0e0f25852f564168f8b796be WatchSource:0}: Error finding container e282f4b8bfd50f1b751675292338850c5724784d0e0f25852f564168f8b796be: Status 404 returned error can't find the container with id e282f4b8bfd50f1b751675292338850c5724784d0e0f25852f564168f8b796be
Nov 03 17:34:24.732968 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:24.732930    4493 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t4sns_c1a0388c-1177-45a8-8eb8-92ef23d8f0bf/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:24.933115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:24.933070    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerStarted","Data":"e282f4b8bfd50f1b751675292338850c5724784d0e0f25852f564168f8b796be"}
Nov 03 17:34:24.954171 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:24.954135    4493 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-t4sns_c1a0388c-1177-45a8-8eb8-92ef23d8f0bf/konnectivity-agent/0.log" limit=1024
Nov 03 17:34:26.943210 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:26.943170    4493 generic.go:356] "Generic (PLEG): container finished" podID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerID="30290a9b437194b5e54aa2ebe7953d1d68cd4d14c3910b435cffde9bf13856bc" exitCode=0
Nov 03 17:34:26.943714 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:26.943240    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"30290a9b437194b5e54aa2ebe7953d1d68cd4d14c3910b435cffde9bf13856bc"}
Nov 03 17:34:30.960984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:30.960911    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerStarted","Data":"50651940f56df0c298906be6286be38c9af66d490d59d2661a546d2a0ba90107"}
Nov 03 17:34:30.960984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:30.960973    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerStarted","Data":"952d2826f4526e90ca552337deb020e24bcaddc354eba8a97fd6fed563fa431d"}
Nov 03 17:34:33.976679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:33.976645    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerStarted","Data":"c978cd49661730cbc802fb3613da4031fd46c77f3587d9f4e45fc9b552427e14"}
Nov 03 17:34:33.977061 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:33.976768    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerStarted","Data":"4c61324482127d7bcd81494f1d83cb9440bceea79fd01655899ea960e70c27f3"}
Nov 03 17:34:34.983062 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:34.983028    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerStarted","Data":"1bec4d0eedda0557a39914262b6c2b70f418d197e18699a9baeaaa4d083b0e69"}
Nov 03 17:34:34.983655 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:34.983090    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerStarted","Data":"f91c43683475615c6305b546dc39f045a7ca6ebba16f6c60a043ca5cf4f413a2"}
Nov 03 17:34:35.276692 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:35.276589    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.636828195 podStartE2EDuration="12.276575649s" totalImagesPullingTime="8.623402363s" totalInitContainerRuntime="16.345091ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:35.27592537 +0000 UTC m=+54.735096357" watchObservedRunningTime="2026-11-03 17:34:35.276575649 +0000 UTC m=+54.735746634"
Nov 03 17:34:36.991621 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:34:36.991584    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:34:39.166344 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:39.166310    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:39.426377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:39.426302    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:39.426377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:39.426364    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:42.851181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:42.851145    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:42.870978 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:42.870943    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:43.025814 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:43.025787    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:34:59.430928 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:59.430894    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:34:59.434686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:34:59.434660    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64477db88b-vv6ll"
Nov 03 17:35:06.722509 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:06.722477    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fssvd-2gvm4"]
Nov 03 17:35:06.723028 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:06.722812    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-fssvd-2gvm4" podUID="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" containerName="insights-gathering" containerID="cri-o://4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee" gracePeriod=30
Nov 03 17:35:06.967096 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:06.967062    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:35:06.992062 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:35:06.991992    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:35:07.013522 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.013492    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\") pod \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") "
Nov 03 17:35:07.013622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.013544    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\" (UniqueName: \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\") pod \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") "
Nov 03 17:35:07.013622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.013570    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\") pod \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") "
Nov 03 17:35:07.013622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.013618    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\") pod \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\" (UID: \"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9\") "
Nov 03 17:35:07.013974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.013949    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle" pod "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" (UID: "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:07.014055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.013973    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path" pod "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" (UID: "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:35:07.014055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.013979    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle" pod "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" (UID: "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:07.015905 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.015866    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w" pod "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" (UID: "d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9"). InnerVolumeSpecName "kube-api-access-b5l8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:07.099356 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.099324    4493 generic.go:356] "Generic (PLEG): container finished" podID="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" containerID="4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee" exitCode=2
Nov 03 17:35:07.099514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.099367    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fssvd-2gvm4"
Nov 03 17:35:07.099514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.099362    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fssvd-2gvm4" event={"ID":"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9","Type":"ContainerDied","Data":"4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee"}
Nov 03 17:35:07.099514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.099430    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fssvd-2gvm4" event={"ID":"d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9","Type":"ContainerDied","Data":"62a07abd2e0db0750661e6865b8eeac22786dfd12548d5721128645c467103ab"}
Nov 03 17:35:07.099514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.099457    4493 scope.go:122] "RemoveContainer" containerID="4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee"
Nov 03 17:35:07.108182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.108161    4493 scope.go:122] "RemoveContainer" containerID="4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee"
Nov 03 17:35:07.108471 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:35:07.108452    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee\": container with ID starting with 4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee not found: ID does not exist" containerID="4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee"
Nov 03 17:35:07.108547 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.108484    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee"} err="failed to get container status \"4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee\": rpc error: code = NotFound desc = could not find container \"4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee\": container with ID starting with 4417e45e36d3e73c4feadc41497f9554780e56de7814e19202dc074f58007bee not found: ID does not exist"
Nov 03 17:35:07.114952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.114934    4493 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-service-ca-bundle\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:07.115028 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.114956    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5l8w\" (UniqueName: \"kubernetes.io/projected/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-kube-api-access-b5l8w\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:07.115028 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.114970    4493 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-trusted-ca-bundle\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:07.115028 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.114983    4493 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9-archives-path\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:07.267961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.267866    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fssvd-2gvm4"]
Nov 03 17:35:07.274220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:07.274189    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-fssvd-2gvm4"]
Nov 03 17:35:08.709896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:08.709833    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" path="/var/lib/kubelet/pods/d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9/volumes"
Nov 03 17:35:13.896119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:13.896084    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:35:13.896650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:13.896414    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="prometheus" containerID="cri-o://952d2826f4526e90ca552337deb020e24bcaddc354eba8a97fd6fed563fa431d" gracePeriod=600
Nov 03 17:35:13.896650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:13.896460    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="thanos-sidecar" containerID="cri-o://4c61324482127d7bcd81494f1d83cb9440bceea79fd01655899ea960e70c27f3" gracePeriod=600
Nov 03 17:35:13.896650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:13.896495    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="config-reloader" containerID="cri-o://50651940f56df0c298906be6286be38c9af66d490d59d2661a546d2a0ba90107" gracePeriod=600
Nov 03 17:35:13.896650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:13.896509    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-web" containerID="cri-o://c978cd49661730cbc802fb3613da4031fd46c77f3587d9f4e45fc9b552427e14" gracePeriod=600
Nov 03 17:35:13.896650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:13.896457    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy" containerID="cri-o://f91c43683475615c6305b546dc39f045a7ca6ebba16f6c60a043ca5cf4f413a2" gracePeriod=600
Nov 03 17:35:13.896650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:13.896509    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1bec4d0eedda0557a39914262b6c2b70f418d197e18699a9baeaaa4d083b0e69" gracePeriod=600
Nov 03 17:35:14.128486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128448    4493 generic.go:356] "Generic (PLEG): container finished" podID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerID="1bec4d0eedda0557a39914262b6c2b70f418d197e18699a9baeaaa4d083b0e69" exitCode=0
Nov 03 17:35:14.128486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128472    4493 generic.go:356] "Generic (PLEG): container finished" podID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerID="f91c43683475615c6305b546dc39f045a7ca6ebba16f6c60a043ca5cf4f413a2" exitCode=0
Nov 03 17:35:14.128486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128478    4493 generic.go:356] "Generic (PLEG): container finished" podID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerID="c978cd49661730cbc802fb3613da4031fd46c77f3587d9f4e45fc9b552427e14" exitCode=0
Nov 03 17:35:14.128486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128483    4493 generic.go:356] "Generic (PLEG): container finished" podID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerID="4c61324482127d7bcd81494f1d83cb9440bceea79fd01655899ea960e70c27f3" exitCode=0
Nov 03 17:35:14.128486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128488    4493 generic.go:356] "Generic (PLEG): container finished" podID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerID="50651940f56df0c298906be6286be38c9af66d490d59d2661a546d2a0ba90107" exitCode=0
Nov 03 17:35:14.128486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128496    4493 generic.go:356] "Generic (PLEG): container finished" podID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerID="952d2826f4526e90ca552337deb020e24bcaddc354eba8a97fd6fed563fa431d" exitCode=0
Nov 03 17:35:14.128797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128514    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"1bec4d0eedda0557a39914262b6c2b70f418d197e18699a9baeaaa4d083b0e69"}
Nov 03 17:35:14.128797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128580    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"f91c43683475615c6305b546dc39f045a7ca6ebba16f6c60a043ca5cf4f413a2"}
Nov 03 17:35:14.128797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128606    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"c978cd49661730cbc802fb3613da4031fd46c77f3587d9f4e45fc9b552427e14"}
Nov 03 17:35:14.128797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128628    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"4c61324482127d7bcd81494f1d83cb9440bceea79fd01655899ea960e70c27f3"}
Nov 03 17:35:14.128797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128651    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"50651940f56df0c298906be6286be38c9af66d490d59d2661a546d2a0ba90107"}
Nov 03 17:35:14.128797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.128673    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"952d2826f4526e90ca552337deb020e24bcaddc354eba8a97fd6fed563fa431d"}
Nov 03 17:35:14.142502 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.142481    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:14.166463 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166412    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-db\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166463 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166444    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-kubelet-serving-ca-bundle\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166592 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166464    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-metrics-client-certs\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166592 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166501    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166592 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166533    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-0\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166592 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166564    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-trusted-ca-bundle\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166595    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-kube-api-access-df8tm\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-kube-api-access-df8tm\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166631    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166658    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config-out\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166688    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-kube-rbac-proxy\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166717    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-metrics-client-ca\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.166786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166759    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-serving-certs-ca-bundle\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166797    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166830    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-grpc-tls\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166851    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-web-config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-web-config\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166917    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-tls-assets\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166955    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-1\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166996    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-tls\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.166994    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-kubelet-serving-ca-bundle" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.167032    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-thanos-prometheus-http-client-file\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.167069    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-2\") pod \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\" (UID: \"fdb49ab7-ea8a-4d38-b284-98bfc251aafe\") "
Nov 03 17:35:14.167110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.167086    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-trusted-ca-bundle" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:14.167609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.167288    4493 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.167609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.167311    4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.168010 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.167937    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-2" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:14.168626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.168578    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-1" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:14.169976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.169629    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-metrics-client-ca" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:14.170067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.169971    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.170067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.170017    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-0" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:14.170343 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.170310    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-serving-certs-ca-bundle" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:35:14.170537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.170492    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-db" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:35:14.170630 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.170588    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-metrics-client-certs" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.171156 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.171130    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-kube-api-access-df8tm" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "kube-api-access-df8tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:14.171262 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.171145    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config-out" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:35:14.171516 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.171493    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-grpc-tls" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.171724 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.171693    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-tls-assets" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:35:14.171724 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.171705    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-tls" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.171834 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.171778    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.172362 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.172341    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-kube-rbac-proxy" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.172815 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.172794    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-thanos-prometheus-http-client-file" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.172896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.172820    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-thanos-sidecar-tls" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.180676 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.180658    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-web-config" pod "fdb49ab7-ea8a-4d38-b284-98bfc251aafe" (UID: "fdb49ab7-ea8a-4d38-b284-98bfc251aafe"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:35:14.268193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268164    4493 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-grpc-tls\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268187    4493 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-web-config\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268198    4493 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-tls-assets\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268207    4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268216    4493 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268225    4493 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268234    4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268243    4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-db\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268251    4493 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-metrics-client-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268260    4493 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268268    4493 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268276    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df8tm\" (UniqueName: \"kubernetes.io/projected/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-kube-api-access-df8tm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268284    4493 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268292    4493 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-config-out\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268300    4493 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-kube-rbac-proxy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268309    4493 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-metrics-client-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268317    4493 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:14.268395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:14.268326    4493 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdb49ab7-ea8a-4d38-b284-98bfc251aafe-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:35:15.135387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.135306    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdb49ab7-ea8a-4d38-b284-98bfc251aafe","Type":"ContainerDied","Data":"e282f4b8bfd50f1b751675292338850c5724784d0e0f25852f564168f8b796be"}
Nov 03 17:35:15.135387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.135363    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.135387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.135382    4493 scope.go:122] "RemoveContainer" containerID="1bec4d0eedda0557a39914262b6c2b70f418d197e18699a9baeaaa4d083b0e69"
Nov 03 17:35:15.142692 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.142671    4493 scope.go:122] "RemoveContainer" containerID="f91c43683475615c6305b546dc39f045a7ca6ebba16f6c60a043ca5cf4f413a2"
Nov 03 17:35:15.149311 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.149296    4493 scope.go:122] "RemoveContainer" containerID="c978cd49661730cbc802fb3613da4031fd46c77f3587d9f4e45fc9b552427e14"
Nov 03 17:35:15.155711 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.155694    4493 scope.go:122] "RemoveContainer" containerID="4c61324482127d7bcd81494f1d83cb9440bceea79fd01655899ea960e70c27f3"
Nov 03 17:35:15.161842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.161826    4493 scope.go:122] "RemoveContainer" containerID="50651940f56df0c298906be6286be38c9af66d490d59d2661a546d2a0ba90107"
Nov 03 17:35:15.168201 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.168182    4493 scope.go:122] "RemoveContainer" containerID="952d2826f4526e90ca552337deb020e24bcaddc354eba8a97fd6fed563fa431d"
Nov 03 17:35:15.174916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.174895    4493 scope.go:122] "RemoveContainer" containerID="30290a9b437194b5e54aa2ebe7953d1d68cd4d14c3910b435cffde9bf13856bc"
Nov 03 17:35:15.310543 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.310514    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:35:15.360816 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.360788    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:35:15.361368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361351    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:35:15.361585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361573    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="init-config-reloader"
Nov 03 17:35:15.361622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361587    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="init-config-reloader"
Nov 03 17:35:15.361622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361597    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-web"
Nov 03 17:35:15.361622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361602    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-web"
Nov 03 17:35:15.361622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361610    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="config-reloader"
Nov 03 17:35:15.361622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361615    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="config-reloader"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361625    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" containerName="insights-gathering"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361630    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" containerName="insights-gathering"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361636    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="prometheus"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361641    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="prometheus"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361646    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="thanos-sidecar"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361651    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="thanos-sidecar"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361659    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361662    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361676    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-thanos"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361680    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-thanos"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361716    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-thanos"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361722    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy-web"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361728    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="config-reloader"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361735    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bef3d2-aed3-4a2a-bf78-6f6a2febe2f9" containerName="insights-gathering"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361740    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="prometheus"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361743    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="thanos-sidecar"
Nov 03 17:35:15.361764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.361749    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" containerName="kube-rbac-proxy"
Nov 03 17:35:15.365052 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.365035    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.419976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.419917    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 17:35:15.420082 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.420013    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 17:35:15.420741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.420724    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 17:35:15.421024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.421001    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 17:35:15.421097 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.421052    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8f8m48m7s3jj2\""
Nov 03 17:35:15.421161 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.421105    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 17:35:15.421161 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.421156    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t8sg4\""
Nov 03 17:35:15.421234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.421161    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 17:35:15.421234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.421168    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 17:35:15.424710 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.424693    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 17:35:15.425132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.425117    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 17:35:15.425170 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.425150    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 17:35:15.425483 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.425465    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 17:35:15.425517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.425499    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 17:35:15.476547 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476522    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 17:35:15.476671 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476657    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 17:35:15.476751 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476678    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-config\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.476751 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476700    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.476751 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476718    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.476845 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476788    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07a7b2c5-bb90-42d6-8529-0a9d4444d356-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.476845 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476820    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.476940 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476853    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.476940 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476900    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-web-config\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476949    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.476966    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477078 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477018    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07a7b2c5-bb90-42d6-8529-0a9d4444d356-config-out\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477078 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477038    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477078 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477053    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477078 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477068    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477084    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477144    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477196    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477217    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477262    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477296    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9jq\" (UniqueName: \"kubernetes.io/projected/07a7b2c5-bb90-42d6-8529-0a9d4444d356-kube-api-access-zt9jq\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.477415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.477383    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578429 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578401    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07a7b2c5-bb90-42d6-8529-0a9d4444d356-config-out\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578429 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578433    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578447    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578465    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578490    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578515    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578547    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578575    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578596    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578611    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9jq\" (UniqueName: \"kubernetes.io/projected/07a7b2c5-bb90-42d6-8529-0a9d4444d356-kube-api-access-zt9jq\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578633    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578665    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-config\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578687    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.578799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.578710    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.579146 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.579109    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07a7b2c5-bb90-42d6-8529-0a9d4444d356-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.579193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.579168    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.579240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.579208    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.579287 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.579243    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-web-config\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.579337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.579282    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.579337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.579326    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.579703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.579678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.580784 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.580752    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.580908 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.580865    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.581473 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.581438    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.584127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.582164    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07a7b2c5-bb90-42d6-8529-0a9d4444d356-config-out\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.584127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.583230    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.584127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.583685    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.584127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.583895    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.584703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.584676    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.584950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.584925    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.585711 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.585689    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07a7b2c5-bb90-42d6-8529-0a9d4444d356-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.587819 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.587790    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.587945 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.587833    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.588740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.588715    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.589243 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.589219    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-config\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.589506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.589488    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.589663 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.589642    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-web-config\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.589720 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.589691    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07a7b2c5-bb90-42d6-8529-0a9d4444d356-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.590137 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.590116    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07a7b2c5-bb90-42d6-8529-0a9d4444d356-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.611117 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.611096    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:35:15.647711 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.647683    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9jq\" (UniqueName: \"kubernetes.io/projected/07a7b2c5-bb90-42d6-8529-0a9d4444d356-kube-api-access-zt9jq\") pod \"prometheus-k8s-0\" (UID: \"07a7b2c5-bb90-42d6-8529-0a9d4444d356\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:15.680786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:15.680711    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:16.130737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:16.130634    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 17:35:16.133546 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:35:16.133509    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a7b2c5_bb90_42d6_8529_0a9d4444d356.slice/crio-0abc32710b3318063b33073f16d944af2c3a50b223b8748f3807a9ddc232ed6f WatchSource:0}: Error finding container 0abc32710b3318063b33073f16d944af2c3a50b223b8748f3807a9ddc232ed6f: Status 404 returned error can't find the container with id 0abc32710b3318063b33073f16d944af2c3a50b223b8748f3807a9ddc232ed6f
Nov 03 17:35:16.140938 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:16.140908    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerStarted","Data":"0abc32710b3318063b33073f16d944af2c3a50b223b8748f3807a9ddc232ed6f"}
Nov 03 17:35:16.709831 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:16.709796    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb49ab7-ea8a-4d38-b284-98bfc251aafe" path="/var/lib/kubelet/pods/fdb49ab7-ea8a-4d38-b284-98bfc251aafe/volumes"
Nov 03 17:35:17.145756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:17.145724    4493 generic.go:356] "Generic (PLEG): container finished" podID="07a7b2c5-bb90-42d6-8529-0a9d4444d356" containerID="4c1ead399e16d23dd1799ae928855249417bbc70b4b349a55894a3eb64e941a2" exitCode=0
Nov 03 17:35:17.146265 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:17.145787    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerDied","Data":"4c1ead399e16d23dd1799ae928855249417bbc70b4b349a55894a3eb64e941a2"}
Nov 03 17:35:18.153156 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:18.153122    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerStarted","Data":"998f9d79edd5be6cb6d8fbb6862659a301ea605a17a5b3e5a77453c2b9a12160"}
Nov 03 17:35:18.153579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:18.153185    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerStarted","Data":"2fd3283d8cb3a9b966be22d09292c2a4b2b2c0296d387be7f2d9849089038796"}
Nov 03 17:35:18.153579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:18.153211    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerStarted","Data":"fda6886afac10c1f2a77ceeba9646c2dd864bcc53162466d61240b70c935f8b6"}
Nov 03 17:35:18.153579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:18.153235    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerStarted","Data":"0b0362eed7cbbad5e69ed569b01a2f3e2169da74d594bc73a333d66241460e10"}
Nov 03 17:35:18.153579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:18.153259    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerStarted","Data":"a4d9fea4868ca9521f231760fda93552cfd287c570cce61002f2ebb31b8e1d70"}
Nov 03 17:35:18.153579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:18.153281    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07a7b2c5-bb90-42d6-8529-0a9d4444d356","Type":"ContainerStarted","Data":"bc5754e6ee323381b0efa0d79facce3bced5e7c4bad2e4dddcf0c300e7f58895"}
Nov 03 17:35:18.291795 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:18.291737    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.274883712 podStartE2EDuration="3.291719402s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.83569ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:18.290789825 +0000 UTC m=+97.749960835" watchObservedRunningTime="2026-11-03 17:35:18.291719402 +0000 UTC m=+97.750890384"
Nov 03 17:35:20.681207 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:35:20.681174    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:35:36.990937 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:35:36.990902    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:36:06.989748 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:36:06.989674    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:36:15.681587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:36:15.681550    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:36:15.697263 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:36:15.697240    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:36:16.352580 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:36:16.352555    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 17:36:36.991542 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:36:36.991500    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:37:06.990845 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:37:06.990812    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:37:36.990371 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:37:36.990327    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:38:06.990818 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:38:06.990779    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:38:36.990956 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:38:36.990910    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:39:06.990893 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:39:06.990791    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:39:36.990681 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:39:36.990640    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:40:06.990149 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:40:06.990108    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:40:23.763433 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.763351    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"]
Nov 03 17:40:23.766706 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.766687    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.829435 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.829400    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.829607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.829447    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv7gq\" (UniqueName: \"kubernetes.io/projected/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-kube-api-access-bv7gq\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.829607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.829486    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.829607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.829544    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.829607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.829584    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.829758 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.829682    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.893598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.893570    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 17:40:23.893761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.893605    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 17:40:23.893761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.893625    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 17:40:23.893761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.893673    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 17:40:23.893761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.893690    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 17:40:23.893933 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.893765    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kqpc6\""
Nov 03 17:40:23.930699 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.930676    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.930839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.930710    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv7gq\" (UniqueName: \"kubernetes.io/projected/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-kube-api-access-bv7gq\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.930839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.930729    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.930839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.930743    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.930839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.930766    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.931062 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.931020    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.931371 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.931349    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.933658 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.933637    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.933767 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.933731    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.933945 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.933926    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.934068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.934052    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:23.994745 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:23.994721    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv7gq\" (UniqueName: \"kubernetes.io/projected/6fc731c3-5369-42b3-9bcf-c8c2e834cb6d-kube-api-access-bv7gq\") pod \"opendatahub-operator-controller-manager-57dbb86d66-22njt\" (UID: \"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:24.025133 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:24.025066    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"]
Nov 03 17:40:24.081099 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:24.081072    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:24.575269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:24.575193    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"]
Nov 03 17:40:24.578076 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:40:24.578044    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc731c3_5369_42b3_9bcf_c8c2e834cb6d.slice/crio-5afbf510c5985888dad63af2911a57f55d60b4f6bd456d2b8ad0d274d6df99c9 WatchSource:0}: Error finding container 5afbf510c5985888dad63af2911a57f55d60b4f6bd456d2b8ad0d274d6df99c9: Status 404 returned error can't find the container with id 5afbf510c5985888dad63af2911a57f55d60b4f6bd456d2b8ad0d274d6df99c9
Nov 03 17:40:24.579641 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:24.579624    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:40:25.128961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:25.128920    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt" event={"ID":"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d","Type":"ContainerStarted","Data":"5afbf510c5985888dad63af2911a57f55d60b4f6bd456d2b8ad0d274d6df99c9"}
Nov 03 17:40:28.142675 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:28.142645    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt" event={"ID":"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d","Type":"ContainerStarted","Data":"12f387bd396b585481ed46ec1ac74dc662e5dc6d919c1e85b2180aaf87477abc"}
Nov 03 17:40:29.148016 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:29.147973    4493 generic.go:356] "Generic (PLEG): container finished" podID="6fc731c3-5369-42b3-9bcf-c8c2e834cb6d" containerID="12f387bd396b585481ed46ec1ac74dc662e5dc6d919c1e85b2180aaf87477abc" exitCode=0
Nov 03 17:40:29.148587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:29.148025    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt" event={"ID":"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d","Type":"ContainerDied","Data":"12f387bd396b585481ed46ec1ac74dc662e5dc6d919c1e85b2180aaf87477abc"}
Nov 03 17:40:30.153325 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:30.153290    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt" event={"ID":"6fc731c3-5369-42b3-9bcf-c8c2e834cb6d","Type":"ContainerStarted","Data":"7c672d6e34a23a836a054bcbf0c525121499fb9b1ebefd524ebd2c1ff879a683"}
Nov 03 17:40:30.153725 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:30.153425    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:30.403357 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:30.402732    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt" podStartSLOduration=2.687857303 podStartE2EDuration="7.402712704s" totalImagesPullingTime="4.219299668s" totalInitContainerRuntime="495.555733ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:30.352114789 +0000 UTC m=+409.811285765" watchObservedRunningTime="2026-11-03 17:40:30.402712704 +0000 UTC m=+409.861883693"
Nov 03 17:40:36.990769 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:40:36.990728    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:40:38.272577 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.272541    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"]
Nov 03 17:40:38.280340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.280313    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.343453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.343412    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ppz\" (UniqueName: \"kubernetes.io/projected/842046a2-114c-493b-bc33-8233ccb88806-kube-api-access-s4ppz\") pod \"cert-manager-cainjector-5c57bbd4fc-bqcg7\" (UID: \"842046a2-114c-493b-bc33-8233ccb88806\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.343639 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.343462    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/842046a2-114c-493b-bc33-8233ccb88806-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-bqcg7\" (UID: \"842046a2-114c-493b-bc33-8233ccb88806\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.344330 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.344304    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"]
Nov 03 17:40:38.393955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.393916    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 17:40:38.393955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.393933    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-f2ztw\""
Nov 03 17:40:38.393955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.393932    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 17:40:38.443990 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.443957    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ppz\" (UniqueName: \"kubernetes.io/projected/842046a2-114c-493b-bc33-8233ccb88806-kube-api-access-s4ppz\") pod \"cert-manager-cainjector-5c57bbd4fc-bqcg7\" (UID: \"842046a2-114c-493b-bc33-8233ccb88806\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.443990 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.443990    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/842046a2-114c-493b-bc33-8233ccb88806-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-bqcg7\" (UID: \"842046a2-114c-493b-bc33-8233ccb88806\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.514023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.513981    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/842046a2-114c-493b-bc33-8233ccb88806-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-bqcg7\" (UID: \"842046a2-114c-493b-bc33-8233ccb88806\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.514165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.514089    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ppz\" (UniqueName: \"kubernetes.io/projected/842046a2-114c-493b-bc33-8233ccb88806-kube-api-access-s4ppz\") pod \"cert-manager-cainjector-5c57bbd4fc-bqcg7\" (UID: \"842046a2-114c-493b-bc33-8233ccb88806\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.604855 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.604776    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"
Nov 03 17:40:38.978743 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:38.978716    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7"]
Nov 03 17:40:38.983370 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:40:38.983345    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod842046a2_114c_493b_bc33_8233ccb88806.slice/crio-22c6e63dc8c19a49189e0e7f238933ebbb5ddf167fbb99269feeb90025e63e93 WatchSource:0}: Error finding container 22c6e63dc8c19a49189e0e7f238933ebbb5ddf167fbb99269feeb90025e63e93: Status 404 returned error can't find the container with id 22c6e63dc8c19a49189e0e7f238933ebbb5ddf167fbb99269feeb90025e63e93
Nov 03 17:40:39.184214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:39.184185    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7" event={"ID":"842046a2-114c-493b-bc33-8233ccb88806","Type":"ContainerStarted","Data":"22c6e63dc8c19a49189e0e7f238933ebbb5ddf167fbb99269feeb90025e63e93"}
Nov 03 17:40:41.160218 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:41.160183    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-22njt"
Nov 03 17:40:43.199138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:43.199060    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7" event={"ID":"842046a2-114c-493b-bc33-8233ccb88806","Type":"ContainerStarted","Data":"c735b98326439a08fe5c206ba947096d774949d9f92482ce5661c030a753a81e"}
Nov 03 17:40:43.324024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:43.323972    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-bqcg7" podStartSLOduration=1.368276766 podStartE2EDuration="5.323959215s" totalImagesPullingTime="3.955682449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:43.323913041 +0000 UTC m=+422.783084025" watchObservedRunningTime="2026-11-03 17:40:43.323959215 +0000 UTC m=+422.783130181"
Nov 03 17:40:46.455431 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.455395    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"]
Nov 03 17:40:46.457643 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.457617    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:46.563488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.563448    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 17:40:46.563488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.563481    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 17:40:46.563488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.563502    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-snz9x\""
Nov 03 17:40:46.564514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.564495    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"]
Nov 03 17:40:46.615067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.615033    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-x8bb9"]
Nov 03 17:40:46.616251 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.616232    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5nvxk\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:46.616315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.616260    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwxz\" (UniqueName: \"kubernetes.io/projected/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-kube-api-access-lpwxz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5nvxk\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:46.617335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.617321    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:46.648496 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.648469    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-x8bb9"]
Nov 03 17:40:46.672530 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.672506    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jrq9j\""
Nov 03 17:40:46.716647 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.716623    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/544fff99-c798-48ba-8345-5a69d304defc-bound-sa-token\") pod \"cert-manager-597b95cb84-x8bb9\" (UID: \"544fff99-c798-48ba-8345-5a69d304defc\") " pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:46.716775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.716654    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4fgs\" (UniqueName: \"kubernetes.io/projected/544fff99-c798-48ba-8345-5a69d304defc-kube-api-access-j4fgs\") pod \"cert-manager-597b95cb84-x8bb9\" (UID: \"544fff99-c798-48ba-8345-5a69d304defc\") " pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:46.716775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.716749    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5nvxk\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:46.716775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.716771    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwxz\" (UniqueName: \"kubernetes.io/projected/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-kube-api-access-lpwxz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5nvxk\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:46.717105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.717088    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5nvxk\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:46.781419 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.781390    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwxz\" (UniqueName: \"kubernetes.io/projected/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-kube-api-access-lpwxz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5nvxk\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:46.817432 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.817392    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/544fff99-c798-48ba-8345-5a69d304defc-bound-sa-token\") pod \"cert-manager-597b95cb84-x8bb9\" (UID: \"544fff99-c798-48ba-8345-5a69d304defc\") " pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:46.817606 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.817436    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4fgs\" (UniqueName: \"kubernetes.io/projected/544fff99-c798-48ba-8345-5a69d304defc-kube-api-access-j4fgs\") pod \"cert-manager-597b95cb84-x8bb9\" (UID: \"544fff99-c798-48ba-8345-5a69d304defc\") " pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:46.888751 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.888716    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/544fff99-c798-48ba-8345-5a69d304defc-bound-sa-token\") pod \"cert-manager-597b95cb84-x8bb9\" (UID: \"544fff99-c798-48ba-8345-5a69d304defc\") " pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:46.888864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.888793    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4fgs\" (UniqueName: \"kubernetes.io/projected/544fff99-c798-48ba-8345-5a69d304defc-kube-api-access-j4fgs\") pod \"cert-manager-597b95cb84-x8bb9\" (UID: \"544fff99-c798-48ba-8345-5a69d304defc\") " pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:46.927771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:46.927742    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-x8bb9"
Nov 03 17:40:47.072374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:47.072337    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:47.305838 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:47.305770    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-x8bb9"]
Nov 03 17:40:47.308586 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:40:47.308562    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod544fff99_c798_48ba_8345_5a69d304defc.slice/crio-da9ccefe6a3a6be7757ab8d95ba085a2fd110ccb7e917fa4897e7b4186926acd WatchSource:0}: Error finding container da9ccefe6a3a6be7757ab8d95ba085a2fd110ccb7e917fa4897e7b4186926acd: Status 404 returned error can't find the container with id da9ccefe6a3a6be7757ab8d95ba085a2fd110ccb7e917fa4897e7b4186926acd
Nov 03 17:40:47.449278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:47.449248    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"]
Nov 03 17:40:47.452340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:40:47.452315    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f452a8f_3c7e_4530_9f1f_7f3c070f3031.slice/crio-7e4a4454314e791f9882b90a96a4e29bc5f175804936ab95662e37139e2260c4 WatchSource:0}: Error finding container 7e4a4454314e791f9882b90a96a4e29bc5f175804936ab95662e37139e2260c4: Status 404 returned error can't find the container with id 7e4a4454314e791f9882b90a96a4e29bc5f175804936ab95662e37139e2260c4
Nov 03 17:40:48.217568 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:48.217530    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk" event={"ID":"0f452a8f-3c7e-4530-9f1f-7f3c070f3031","Type":"ContainerStarted","Data":"7e4a4454314e791f9882b90a96a4e29bc5f175804936ab95662e37139e2260c4"}
Nov 03 17:40:48.219026 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:48.218998    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-x8bb9" event={"ID":"544fff99-c798-48ba-8345-5a69d304defc","Type":"ContainerStarted","Data":"6cbde85ebb9edc78464e9c8b8a9914ea46a2ae81bcd2ca1deaf12a81ab5ed636"}
Nov 03 17:40:48.219144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:48.219038    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-x8bb9" event={"ID":"544fff99-c798-48ba-8345-5a69d304defc","Type":"ContainerStarted","Data":"da9ccefe6a3a6be7757ab8d95ba085a2fd110ccb7e917fa4897e7b4186926acd"}
Nov 03 17:40:48.343443 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:48.343391    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-x8bb9" podStartSLOduration=2.343379899 podStartE2EDuration="2.343379899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:48.342692826 +0000 UTC m=+427.801863811" watchObservedRunningTime="2026-11-03 17:40:48.343379899 +0000 UTC m=+427.802550884"
Nov 03 17:40:52.236548 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:52.236511    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk" event={"ID":"0f452a8f-3c7e-4530-9f1f-7f3c070f3031","Type":"ContainerStarted","Data":"f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df"}
Nov 03 17:40:52.236966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:52.236617    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:40:52.468242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:40:52.468187    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk" podStartSLOduration=2.065160763 podStartE2EDuration="6.468173135s" totalImagesPullingTime="4.403012372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:52.468109536 +0000 UTC m=+431.927280521" watchObservedRunningTime="2026-11-03 17:40:52.468173135 +0000 UTC m=+431.927344110"
Nov 03 17:41:03.242483 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:03.242453    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:41:03.951237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:03.951198    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"]
Nov 03 17:41:03.954609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:03.954587    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.022516 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.022488    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"]
Nov 03 17:41:04.025738 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.025676    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 17:41:04.025960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.025942    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 17:41:04.061847 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.061820    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stjbx\" (UniqueName: \"kubernetes.io/projected/1e569d24-754e-4f20-9448-29cb410869d6-kube-api-access-stjbx\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.061973 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.061860    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e569d24-754e-4f20-9448-29cb410869d6-tls-certs\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.061973 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.061951    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e569d24-754e-4f20-9448-29cb410869d6-tmp\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.071105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.071068    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2zrqq\""
Nov 03 17:41:04.098509 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.098483    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 17:41:04.098642 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.098572    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 17:41:04.163157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.163124    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stjbx\" (UniqueName: \"kubernetes.io/projected/1e569d24-754e-4f20-9448-29cb410869d6-kube-api-access-stjbx\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.163305 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.163166    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e569d24-754e-4f20-9448-29cb410869d6-tls-certs\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.163305 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.163182    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e569d24-754e-4f20-9448-29cb410869d6-tmp\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.165550 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.165525    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e569d24-754e-4f20-9448-29cb410869d6-tmp\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.165710 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.165689    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e569d24-754e-4f20-9448-29cb410869d6-tls-certs\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.226780 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.226752    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stjbx\" (UniqueName: \"kubernetes.io/projected/1e569d24-754e-4f20-9448-29cb410869d6-kube-api-access-stjbx\") pod \"kube-auth-proxy-7c8f76b86f-6w6hr\" (UID: \"1e569d24-754e-4f20-9448-29cb410869d6\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.265691 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.265660    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"
Nov 03 17:41:04.651921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:04.651801    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr"]
Nov 03 17:41:04.654703 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:41:04.654677    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e569d24_754e_4f20_9448_29cb410869d6.slice/crio-d42ef9fd07561721fbdcc23467a675984543559d4d3ebda60f48a65bd0de2940 WatchSource:0}: Error finding container d42ef9fd07561721fbdcc23467a675984543559d4d3ebda60f48a65bd0de2940: Status 404 returned error can't find the container with id d42ef9fd07561721fbdcc23467a675984543559d4d3ebda60f48a65bd0de2940
Nov 03 17:41:05.287844 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:05.287809    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr" event={"ID":"1e569d24-754e-4f20-9448-29cb410869d6","Type":"ContainerStarted","Data":"d42ef9fd07561721fbdcc23467a675984543559d4d3ebda60f48a65bd0de2940"}
Nov 03 17:41:06.668517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.668477    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"]
Nov 03 17:41:06.674960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.674940    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.738825 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.738797    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"]
Nov 03 17:41:06.786331 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.786300    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/abaafca4-8ecb-4e52-a405-38571682a361-config-file\") pod \"limitador-limitador-74dfcfb894-rmjxc\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.786476 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.786390    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvvp\" (UniqueName: \"kubernetes.io/projected/abaafca4-8ecb-4e52-a405-38571682a361-kube-api-access-2tvvp\") pod \"limitador-limitador-74dfcfb894-rmjxc\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.787063 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.787038    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pknc5\""
Nov 03 17:41:06.787173 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.787090    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 17:41:06.887482 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.887450    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvvp\" (UniqueName: \"kubernetes.io/projected/abaafca4-8ecb-4e52-a405-38571682a361-kube-api-access-2tvvp\") pod \"limitador-limitador-74dfcfb894-rmjxc\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.887661 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.887555    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/abaafca4-8ecb-4e52-a405-38571682a361-config-file\") pod \"limitador-limitador-74dfcfb894-rmjxc\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.888539 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.888248    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/abaafca4-8ecb-4e52-a405-38571682a361-config-file\") pod \"limitador-limitador-74dfcfb894-rmjxc\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.920503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.920424    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"]
Nov 03 17:41:06.952757 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.952730    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvvp\" (UniqueName: \"kubernetes.io/projected/abaafca4-8ecb-4e52-a405-38571682a361-kube-api-access-2tvvp\") pod \"limitador-limitador-74dfcfb894-rmjxc\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.989907 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:06.989858    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:06.993951 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:41:06.993914    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:41:07.377233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:07.377211    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"]
Nov 03 17:41:07.379941 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:41:07.379910    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabaafca4_8ecb_4e52_a405_38571682a361.slice/crio-694e36423c1032ab2775a537c925d371690943f9f19ac49bf77cae0c17e8b981 WatchSource:0}: Error finding container 694e36423c1032ab2775a537c925d371690943f9f19ac49bf77cae0c17e8b981: Status 404 returned error can't find the container with id 694e36423c1032ab2775a537c925d371690943f9f19ac49bf77cae0c17e8b981
Nov 03 17:41:08.302636 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:08.302602    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc" event={"ID":"abaafca4-8ecb-4e52-a405-38571682a361","Type":"ContainerStarted","Data":"694e36423c1032ab2775a537c925d371690943f9f19ac49bf77cae0c17e8b981"}
Nov 03 17:41:08.304076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:08.304042    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr" event={"ID":"1e569d24-754e-4f20-9448-29cb410869d6","Type":"ContainerStarted","Data":"e5a8ff7a316cf831f4bef2e0e37cdfab557e9cd27adf6b11e6cbf955c14234f3"}
Nov 03 17:41:08.424042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:08.423989    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-6w6hr" podStartSLOduration=2.539414459 podStartE2EDuration="5.423974634s" totalImagesPullingTime="2.884560175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:08.42347759 +0000 UTC m=+447.882648576" watchObservedRunningTime="2026-11-03 17:41:08.423974634 +0000 UTC m=+447.883145618"
Nov 03 17:41:10.314146 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:10.314104    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc" event={"ID":"abaafca4-8ecb-4e52-a405-38571682a361","Type":"ContainerStarted","Data":"8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605"}
Nov 03 17:41:10.314608 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:10.314245    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:11.219528 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.219458    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc" podStartSLOduration=2.431065512 podStartE2EDuration="5.219443642s" totalImagesPullingTime="2.78837813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:10.434655338 +0000 UTC m=+449.893826338" watchObservedRunningTime="2026-11-03 17:41:11.219443642 +0000 UTC m=+450.678614687"
Nov 03 17:41:11.220253 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.220230    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hbbwk"]
Nov 03 17:41:11.223448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.223426    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk"
Nov 03 17:41:11.285530 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.285504    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t8vd7\""
Nov 03 17:41:11.321981 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.321953    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjk6f\" (UniqueName: \"kubernetes.io/projected/3b6311df-8744-4121-b919-2df5aba30d0e-kube-api-access-tjk6f\") pod \"authorino-56c6c9ddd5-hbbwk\" (UID: \"3b6311df-8744-4121-b919-2df5aba30d0e\") " pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk"
Nov 03 17:41:11.335539 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.335514    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hbbwk"]
Nov 03 17:41:11.422712 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.422680    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjk6f\" (UniqueName: \"kubernetes.io/projected/3b6311df-8744-4121-b919-2df5aba30d0e-kube-api-access-tjk6f\") pod \"authorino-56c6c9ddd5-hbbwk\" (UID: \"3b6311df-8744-4121-b919-2df5aba30d0e\") " pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk"
Nov 03 17:41:11.487064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.487000    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjk6f\" (UniqueName: \"kubernetes.io/projected/3b6311df-8744-4121-b919-2df5aba30d0e-kube-api-access-tjk6f\") pod \"authorino-56c6c9ddd5-hbbwk\" (UID: \"3b6311df-8744-4121-b919-2df5aba30d0e\") " pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk"
Nov 03 17:41:11.534077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.534044    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk"
Nov 03 17:41:11.907313 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:11.907231    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hbbwk"]
Nov 03 17:41:11.910463 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:41:11.910424    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b6311df_8744_4121_b919_2df5aba30d0e.slice/crio-d468ae7b8c2f3a4fac41e52ed7ad4d870edf317eb2e5d977958d6632719bee2f WatchSource:0}: Error finding container d468ae7b8c2f3a4fac41e52ed7ad4d870edf317eb2e5d977958d6632719bee2f: Status 404 returned error can't find the container with id d468ae7b8c2f3a4fac41e52ed7ad4d870edf317eb2e5d977958d6632719bee2f
Nov 03 17:41:12.323342 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:12.323305    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk" event={"ID":"3b6311df-8744-4121-b919-2df5aba30d0e","Type":"ContainerStarted","Data":"d468ae7b8c2f3a4fac41e52ed7ad4d870edf317eb2e5d977958d6632719bee2f"}
Nov 03 17:41:15.336490 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:15.336455    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk" event={"ID":"3b6311df-8744-4121-b919-2df5aba30d0e","Type":"ContainerStarted","Data":"c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02"}
Nov 03 17:41:15.479155 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:15.479096    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk" podStartSLOduration=1.902151548 podStartE2EDuration="4.479082627s" totalImagesPullingTime="2.576931079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:15.477935079 +0000 UTC m=+454.937106065" watchObservedRunningTime="2026-11-03 17:41:15.479082627 +0000 UTC m=+454.938253611"
Nov 03 17:41:21.319394 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:21.319359    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:21.606081 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:21.606001    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"]
Nov 03 17:41:21.606269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:21.606248    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk" podUID="0f452a8f-3c7e-4530-9f1f-7f3c070f3031" containerName="manager" containerID="cri-o://f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df" gracePeriod=10
Nov 03 17:41:22.149139 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.149116    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:41:22.209603 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.209574    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-extensions-socket-volume\") pod \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") "
Nov 03 17:41:22.209786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.209614    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-kube-api-access-lpwxz\" (UniqueName: \"kubernetes.io/projected/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-kube-api-access-lpwxz\") pod \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\" (UID: \"0f452a8f-3c7e-4530-9f1f-7f3c070f3031\") "
Nov 03 17:41:22.210142 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.210115    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-extensions-socket-volume" pod "0f452a8f-3c7e-4530-9f1f-7f3c070f3031" (UID: "0f452a8f-3c7e-4530-9f1f-7f3c070f3031"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:41:22.212041 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.212016    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-kube-api-access-lpwxz" pod "0f452a8f-3c7e-4530-9f1f-7f3c070f3031" (UID: "0f452a8f-3c7e-4530-9f1f-7f3c070f3031"). InnerVolumeSpecName "kube-api-access-lpwxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:41:22.310527 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.310446    4493 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-extensions-socket-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:41:22.310527 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.310472    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwxz\" (UniqueName: \"kubernetes.io/projected/0f452a8f-3c7e-4530-9f1f-7f3c070f3031-kube-api-access-lpwxz\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:41:22.364946 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.364910    4493 generic.go:356] "Generic (PLEG): container finished" podID="0f452a8f-3c7e-4530-9f1f-7f3c070f3031" containerID="f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df" exitCode=0
Nov 03 17:41:22.365407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.364964    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"
Nov 03 17:41:22.365407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.364972    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk" event={"ID":"0f452a8f-3c7e-4530-9f1f-7f3c070f3031","Type":"ContainerDied","Data":"f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df"}
Nov 03 17:41:22.365407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.365011    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk" event={"ID":"0f452a8f-3c7e-4530-9f1f-7f3c070f3031","Type":"ContainerDied","Data":"7e4a4454314e791f9882b90a96a4e29bc5f175804936ab95662e37139e2260c4"}
Nov 03 17:41:22.365407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.365033    4493 scope.go:122] "RemoveContainer" containerID="f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df"
Nov 03 17:41:22.374088 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.374065    4493 scope.go:122] "RemoveContainer" containerID="f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df"
Nov 03 17:41:22.374347 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:41:22.374323    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df\": container with ID starting with f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df not found: ID does not exist" containerID="f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df"
Nov 03 17:41:22.374386 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.374357    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df"} err="failed to get container status \"f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df\": rpc error: code = NotFound desc = could not find container \"f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df\": container with ID starting with f99d0e2fe4dd5f25603c0b864c3d224e7ab86882a580ac12db7d4854436391df not found: ID does not exist"
Nov 03 17:41:22.552238 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.552205    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"]
Nov 03 17:41:22.565971 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.565899    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5nvxk"]
Nov 03 17:41:22.710462 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:22.710416    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f452a8f-3c7e-4530-9f1f-7f3c070f3031" path="/var/lib/kubelet/pods/0f452a8f-3c7e-4530-9f1f-7f3c070f3031/volumes"
Nov 03 17:41:24.140411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.140377    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"]
Nov 03 17:41:24.140792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.140698    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0f452a8f-3c7e-4530-9f1f-7f3c070f3031" containerName="manager"
Nov 03 17:41:24.140792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.140708    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f452a8f-3c7e-4530-9f1f-7f3c070f3031" containerName="manager"
Nov 03 17:41:24.140792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.140780    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f452a8f-3c7e-4530-9f1f-7f3c070f3031" containerName="manager"
Nov 03 17:41:24.190158 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.190131    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.205265 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.205237    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"]
Nov 03 17:41:24.225263 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.225235    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcckp\" (UniqueName: \"kubernetes.io/projected/28610c90-9024-4c7a-bbf2-df7310869659-kube-api-access-lcckp\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.225407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.225297    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28610c90-9024-4c7a-bbf2-df7310869659-manager-config\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.225488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.225393    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28610c90-9024-4c7a-bbf2-df7310869659-metrics-certs\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.225525 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.225500    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28610c90-9024-4c7a-bbf2-df7310869659-cert\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.254641 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.254610    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 17:41:24.254641 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.254641    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 17:41:24.254923 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.254905    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 17:41:24.295496 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.295465    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 17:41:24.295496 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.295466    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-mvxkf\""
Nov 03 17:41:24.295678 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.295530    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 17:41:24.326080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.326051    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28610c90-9024-4c7a-bbf2-df7310869659-manager-config\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.326221 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.326091    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28610c90-9024-4c7a-bbf2-df7310869659-metrics-certs\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.326266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.326214    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28610c90-9024-4c7a-bbf2-df7310869659-cert\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.326301 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.326266    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcckp\" (UniqueName: \"kubernetes.io/projected/28610c90-9024-4c7a-bbf2-df7310869659-kube-api-access-lcckp\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.326782 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.326765    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/28610c90-9024-4c7a-bbf2-df7310869659-manager-config\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.328751 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.328732    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28610c90-9024-4c7a-bbf2-df7310869659-metrics-certs\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.328751 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.328740    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28610c90-9024-4c7a-bbf2-df7310869659-cert\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.390929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.390829    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcckp\" (UniqueName: \"kubernetes.io/projected/28610c90-9024-4c7a-bbf2-df7310869659-kube-api-access-lcckp\") pod \"jobset-controller-manager-575fbcf9fb-zd84k\" (UID: \"28610c90-9024-4c7a-bbf2-df7310869659\") " pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.516837 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.516800    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:24.903709 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:24.903667    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"]
Nov 03 17:41:24.905941 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:41:24.905912    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28610c90_9024_4c7a_bbf2_df7310869659.slice/crio-69f61cabc5f1fa16014ce5ba54f87ca61f66c2c260c8ef7df8f0ab4010ba6a6d WatchSource:0}: Error finding container 69f61cabc5f1fa16014ce5ba54f87ca61f66c2c260c8ef7df8f0ab4010ba6a6d: Status 404 returned error can't find the container with id 69f61cabc5f1fa16014ce5ba54f87ca61f66c2c260c8ef7df8f0ab4010ba6a6d
Nov 03 17:41:25.378494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:25.378462    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k" event={"ID":"28610c90-9024-4c7a-bbf2-df7310869659","Type":"ContainerStarted","Data":"69f61cabc5f1fa16014ce5ba54f87ca61f66c2c260c8ef7df8f0ab4010ba6a6d"}
Nov 03 17:41:25.570595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:25.570563    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"]
Nov 03 17:41:25.570784 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:25.570761    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc" podUID="abaafca4-8ecb-4e52-a405-38571682a361" containerName="limitador" containerID="cri-o://8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605" gracePeriod=30
Nov 03 17:41:26.107338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.107313    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:26.143631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.143565    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abaafca4-8ecb-4e52-a405-38571682a361-config-file\" (UniqueName: \"kubernetes.io/configmap/abaafca4-8ecb-4e52-a405-38571682a361-config-file\") pod \"abaafca4-8ecb-4e52-a405-38571682a361\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") "
Nov 03 17:41:26.143631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.143612    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abaafca4-8ecb-4e52-a405-38571682a361-kube-api-access-2tvvp\" (UniqueName: \"kubernetes.io/projected/abaafca4-8ecb-4e52-a405-38571682a361-kube-api-access-2tvvp\") pod \"abaafca4-8ecb-4e52-a405-38571682a361\" (UID: \"abaafca4-8ecb-4e52-a405-38571682a361\") "
Nov 03 17:41:26.143960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.143936    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abaafca4-8ecb-4e52-a405-38571682a361-config-file" pod "abaafca4-8ecb-4e52-a405-38571682a361" (UID: "abaafca4-8ecb-4e52-a405-38571682a361"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:41:26.145769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.145744    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaafca4-8ecb-4e52-a405-38571682a361-kube-api-access-2tvvp" pod "abaafca4-8ecb-4e52-a405-38571682a361" (UID: "abaafca4-8ecb-4e52-a405-38571682a361"). InnerVolumeSpecName "kube-api-access-2tvvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:41:26.244965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.244937    4493 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/abaafca4-8ecb-4e52-a405-38571682a361-config-file\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:41:26.244965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.244966    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tvvp\" (UniqueName: \"kubernetes.io/projected/abaafca4-8ecb-4e52-a405-38571682a361-kube-api-access-2tvvp\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:41:26.383739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.383706    4493 generic.go:356] "Generic (PLEG): container finished" podID="abaafca4-8ecb-4e52-a405-38571682a361" containerID="8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605" exitCode=0
Nov 03 17:41:26.384186 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.383756    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"
Nov 03 17:41:26.384186 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.383749    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc" event={"ID":"abaafca4-8ecb-4e52-a405-38571682a361","Type":"ContainerDied","Data":"8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605"}
Nov 03 17:41:26.384186 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.383899    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-rmjxc" event={"ID":"abaafca4-8ecb-4e52-a405-38571682a361","Type":"ContainerDied","Data":"694e36423c1032ab2775a537c925d371690943f9f19ac49bf77cae0c17e8b981"}
Nov 03 17:41:26.384186 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.383934    4493 scope.go:122] "RemoveContainer" containerID="8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605"
Nov 03 17:41:26.392206 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.392186    4493 scope.go:122] "RemoveContainer" containerID="8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605"
Nov 03 17:41:26.392458 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:41:26.392441    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605\": container with ID starting with 8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605 not found: ID does not exist" containerID="8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605"
Nov 03 17:41:26.392526 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.392466    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605"} err="failed to get container status \"8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605\": rpc error: code = NotFound desc = could not find container \"8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605\": container with ID starting with 8f155b7dcfff675aa25722ad8eba2a0ab77bedd4e992cc62cd9bd793ed9a5605 not found: ID does not exist"
Nov 03 17:41:26.555355 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.555323    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"]
Nov 03 17:41:26.561935 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.561915    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-rmjxc"]
Nov 03 17:41:26.709732 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:26.709703    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abaafca4-8ecb-4e52-a405-38571682a361" path="/var/lib/kubelet/pods/abaafca4-8ecb-4e52-a405-38571682a361/volumes"
Nov 03 17:41:29.398480 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:29.398443    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k" event={"ID":"28610c90-9024-4c7a-bbf2-df7310869659","Type":"ContainerStarted","Data":"bb4608ae43b7c6c9b706a72c6ceb46a7383a492c18e54979e1383505fc318adc"}
Nov 03 17:41:29.398904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:29.398539    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:41:29.522419 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:29.522374    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k" podStartSLOduration=1.2747221180000001 podStartE2EDuration="5.522360812s" totalImagesPullingTime="4.247638694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:29.522018953 +0000 UTC m=+468.981189942" watchObservedRunningTime="2026-11-03 17:41:29.522360812 +0000 UTC m=+468.981531797"
Nov 03 17:41:36.990523 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:41:36.990482    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:41:40.407735 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:41:40.407698    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-575fbcf9fb-zd84k"
Nov 03 17:42:06.991455 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:42:06.991368    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:42:23.761593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.761558    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"]
Nov 03 17:42:23.762080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.761855    4493 cpu_manager.go:414] "containerMap: removing container" podUID="abaafca4-8ecb-4e52-a405-38571682a361" containerName="limitador"
Nov 03 17:42:23.762080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.761866    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abaafca4-8ecb-4e52-a405-38571682a361" containerName="limitador"
Nov 03 17:42:23.762080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.761937    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="abaafca4-8ecb-4e52-a405-38571682a361" containerName="limitador"
Nov 03 17:42:23.763625 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.763608    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.784178 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.784154    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.784275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.784185    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc2kv\" (UniqueName: \"kubernetes.io/projected/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-kube-api-access-dc2kv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.784275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.784218    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.784363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.784344    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.868094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.868059    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 17:42:23.868256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.868110    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 17:42:23.868256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.868121    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 17:42:23.868256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.868159    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 17:42:23.868256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.868220    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2nhn8\""
Nov 03 17:42:23.869055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.869038    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 17:42:23.869633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.869617    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"]
Nov 03 17:42:23.885474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.885435    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.885596 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.885483    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.885596 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.885541    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.885596 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.885560    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc2kv\" (UniqueName: \"kubernetes.io/projected/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-kube-api-access-dc2kv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.885596 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:42:23.885581    4493 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 17:42:23.885803 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:42:23.885634    4493 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 17:42:23.885803 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:42:23.885637    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-tls-cert podName:7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:24.385621692 +0000 UTC m=+523.844792659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-mr6xh" (UID: "7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 17:42:23.885803 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:42:23.885710    4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-proxy-source podName:7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:24.385692538 +0000 UTC m=+523.844863502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-mr6xh" (UID: "7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6") : configmap "usage-tenancy-proxy-source" not found
Nov 03 17:42:23.886167 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.886149    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:23.950464 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:23.950438    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc2kv\" (UniqueName: \"kubernetes.io/projected/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-kube-api-access-dc2kv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:24.388913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:24.388862    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:24.388913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:24.388917    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:24.389749 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:24.389725    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:24.391448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:24.391421    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-mr6xh\" (UID: \"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:24.675094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:24.675021    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:25.048813 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:25.048782    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"]
Nov 03 17:42:25.052146 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:42:25.052118    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2f5f6b_99b5_4be6_a9f1_98b05f468ae6.slice/crio-644e6455289ad6d524fc271795c99d335269e114e77c31a04f421cd9c48aea19 WatchSource:0}: Error finding container 644e6455289ad6d524fc271795c99d335269e114e77c31a04f421cd9c48aea19: Status 404 returned error can't find the container with id 644e6455289ad6d524fc271795c99d335269e114e77c31a04f421cd9c48aea19
Nov 03 17:42:25.617508 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:25.617466    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh" event={"ID":"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6","Type":"ContainerStarted","Data":"644e6455289ad6d524fc271795c99d335269e114e77c31a04f421cd9c48aea19"}
Nov 03 17:42:26.359757 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.359714    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"]
Nov 03 17:42:26.362668 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.362644    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:26.404738 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.404710    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6rc\" (UniqueName: \"kubernetes.io/projected/3e3fbe0d-a31f-420a-b172-fef6fef276d7-kube-api-access-wk6rc\") pod \"llmisvc-controller-manager-869d5dcc5d-zq5x5\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:26.404908 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.404744    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e3fbe0d-a31f-420a-b172-fef6fef276d7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zq5x5\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:26.446977 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.446952    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 17:42:26.467072 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.467051    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7xd65\""
Nov 03 17:42:26.468286 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.468264    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"]
Nov 03 17:42:26.505612 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.505587    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6rc\" (UniqueName: \"kubernetes.io/projected/3e3fbe0d-a31f-420a-b172-fef6fef276d7-kube-api-access-wk6rc\") pod \"llmisvc-controller-manager-869d5dcc5d-zq5x5\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:26.505713 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.505615    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e3fbe0d-a31f-420a-b172-fef6fef276d7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zq5x5\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:26.508031 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.508013    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e3fbe0d-a31f-420a-b172-fef6fef276d7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-zq5x5\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:26.571399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.571369    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6rc\" (UniqueName: \"kubernetes.io/projected/3e3fbe0d-a31f-420a-b172-fef6fef276d7-kube-api-access-wk6rc\") pod \"llmisvc-controller-manager-869d5dcc5d-zq5x5\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:26.674184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:26.674123    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:27.043653 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:27.043621    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"]
Nov 03 17:42:27.046442 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:42:27.046412    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e3fbe0d_a31f_420a_b172_fef6fef276d7.slice/crio-9a96d774d890674deb8015f72910a08c33dfd2e8b9aa87fc799bd103807d52db WatchSource:0}: Error finding container 9a96d774d890674deb8015f72910a08c33dfd2e8b9aa87fc799bd103807d52db: Status 404 returned error can't find the container with id 9a96d774d890674deb8015f72910a08c33dfd2e8b9aa87fc799bd103807d52db
Nov 03 17:42:27.626543 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:27.626504    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5" event={"ID":"3e3fbe0d-a31f-420a-b172-fef6fef276d7","Type":"ContainerStarted","Data":"9a96d774d890674deb8015f72910a08c33dfd2e8b9aa87fc799bd103807d52db"}
Nov 03 17:42:30.650423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:30.650386    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5" event={"ID":"3e3fbe0d-a31f-420a-b172-fef6fef276d7","Type":"ContainerStarted","Data":"1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc"}
Nov 03 17:42:30.650849 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:30.650539    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:42:36.993967 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:42:36.993907    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:42:39.692299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:39.692261    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh" event={"ID":"7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6","Type":"ContainerStarted","Data":"ac540563454d08039c6cf16177ef8230178203a95f6b3b756639cea9812ccf39"}
Nov 03 17:42:39.692695 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:39.692345    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:42:39.815584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:39.815531    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh" podStartSLOduration=3.084568756 podStartE2EDuration="16.815516968s" totalImagesPullingTime="13.730948212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:39.814540159 +0000 UTC m=+539.273711158" watchObservedRunningTime="2026-11-03 17:42:39.815516968 +0000 UTC m=+539.274687953"
Nov 03 17:42:39.815743 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:39.815613    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5" podStartSLOduration=11.030719239 podStartE2EDuration="13.815609668s" totalImagesPullingTime="2.784890429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:30.77365873 +0000 UTC m=+530.232829718" watchObservedRunningTime="2026-11-03 17:42:39.815609668 +0000 UTC m=+539.274780654"
Nov 03 17:42:50.701935 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:42:50.701901    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-mr6xh"
Nov 03 17:43:01.658234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:01.658201    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:43:06.440394 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:06.440361    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jj6bv"]
Nov 03 17:43:06.665600 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:06.665560    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jj6bv"]
Nov 03 17:43:06.665769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:06.665659    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jj6bv"
Nov 03 17:43:06.750217 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:06.750185    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mx8d\" (UniqueName: \"kubernetes.io/projected/b05ba4b7-0164-4e6a-8cdd-b3add04a181f-kube-api-access-7mx8d\") pod \"authorino-6f6944bcdd-jj6bv\" (UID: \"b05ba4b7-0164-4e6a-8cdd-b3add04a181f\") " pod="kuadrant-system/authorino-6f6944bcdd-jj6bv"
Nov 03 17:43:06.836153 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:06.836122    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jj6bv"]
Nov 03 17:43:06.836330 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:43:06.836312    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7mx8d], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-jj6bv" podUID="b05ba4b7-0164-4e6a-8cdd-b3add04a181f"
Nov 03 17:43:06.851384 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:06.851361    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mx8d\" (UniqueName: \"kubernetes.io/projected/b05ba4b7-0164-4e6a-8cdd-b3add04a181f-kube-api-access-7mx8d\") pod \"authorino-6f6944bcdd-jj6bv\" (UID: \"b05ba4b7-0164-4e6a-8cdd-b3add04a181f\") " pod="kuadrant-system/authorino-6f6944bcdd-jj6bv"
Nov 03 17:43:06.915848 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:06.915823    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mx8d\" (UniqueName: \"kubernetes.io/projected/b05ba4b7-0164-4e6a-8cdd-b3add04a181f-kube-api-access-7mx8d\") pod \"authorino-6f6944bcdd-jj6bv\" (UID: \"b05ba4b7-0164-4e6a-8cdd-b3add04a181f\") " pod="kuadrant-system/authorino-6f6944bcdd-jj6bv"
Nov 03 17:43:06.991618 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:43:06.991526    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:43:07.316729 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.316695    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b76779445-xg76x"]
Nov 03 17:43:07.332474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.332450    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.379037 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.379008    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b76779445-xg76x"]
Nov 03 17:43:07.427959 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.427933    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 17:43:07.458141 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.458113    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/654707e4-2300-419f-9d26-151ea79567ab-tls-cert\") pod \"authorino-5b76779445-xg76x\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") " pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.458488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.458184    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjd7s\" (UniqueName: \"kubernetes.io/projected/654707e4-2300-419f-9d26-151ea79567ab-kube-api-access-rjd7s\") pod \"authorino-5b76779445-xg76x\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") " pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.558921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.558895    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjd7s\" (UniqueName: \"kubernetes.io/projected/654707e4-2300-419f-9d26-151ea79567ab-kube-api-access-rjd7s\") pod \"authorino-5b76779445-xg76x\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") " pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.559042 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.558936    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/654707e4-2300-419f-9d26-151ea79567ab-tls-cert\") pod \"authorino-5b76779445-xg76x\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") " pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.561533 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.561512    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/654707e4-2300-419f-9d26-151ea79567ab-tls-cert\") pod \"authorino-5b76779445-xg76x\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") " pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.622587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.622513    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjd7s\" (UniqueName: \"kubernetes.io/projected/654707e4-2300-419f-9d26-151ea79567ab-kube-api-access-rjd7s\") pod \"authorino-5b76779445-xg76x\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") " pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.660214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.660190    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:43:07.797234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.797204    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jj6bv"
Nov 03 17:43:07.823690 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.823667    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jj6bv"
Nov 03 17:43:07.963569 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.963546    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05ba4b7-0164-4e6a-8cdd-b3add04a181f-kube-api-access-7mx8d\" (UniqueName: \"kubernetes.io/projected/b05ba4b7-0164-4e6a-8cdd-b3add04a181f-kube-api-access-7mx8d\") pod \"b05ba4b7-0164-4e6a-8cdd-b3add04a181f\" (UID: \"b05ba4b7-0164-4e6a-8cdd-b3add04a181f\") "
Nov 03 17:43:07.965638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:07.965610    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05ba4b7-0164-4e6a-8cdd-b3add04a181f-kube-api-access-7mx8d" pod "b05ba4b7-0164-4e6a-8cdd-b3add04a181f" (UID: "b05ba4b7-0164-4e6a-8cdd-b3add04a181f"). InnerVolumeSpecName "kube-api-access-7mx8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:08.054246 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:08.054225    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b76779445-xg76x"]
Nov 03 17:43:08.057003 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:43:08.056973    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod654707e4_2300_419f_9d26_151ea79567ab.slice/crio-66005c85d720a0b908d027101b9cc353ffa554f34e350e89d17e9cf0978dea5b WatchSource:0}: Error finding container 66005c85d720a0b908d027101b9cc353ffa554f34e350e89d17e9cf0978dea5b: Status 404 returned error can't find the container with id 66005c85d720a0b908d027101b9cc353ffa554f34e350e89d17e9cf0978dea5b
Nov 03 17:43:08.064138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:08.064110    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mx8d\" (UniqueName: \"kubernetes.io/projected/b05ba4b7-0164-4e6a-8cdd-b3add04a181f-kube-api-access-7mx8d\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:43:08.801685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:08.801645    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b76779445-xg76x" event={"ID":"654707e4-2300-419f-9d26-151ea79567ab","Type":"ContainerStarted","Data":"66005c85d720a0b908d027101b9cc353ffa554f34e350e89d17e9cf0978dea5b"}
Nov 03 17:43:08.801685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:08.801682    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jj6bv"
Nov 03 17:43:09.035249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:09.035164    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jj6bv"]
Nov 03 17:43:09.041524 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:09.041498    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jj6bv"]
Nov 03 17:43:09.807058 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:09.807021    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b76779445-xg76x" event={"ID":"654707e4-2300-419f-9d26-151ea79567ab","Type":"ContainerStarted","Data":"c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21"}
Nov 03 17:43:09.927893 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:09.927826    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b76779445-xg76x" podStartSLOduration=1.696531373 podStartE2EDuration="2.927813331s" totalImagesPullingTime="1.231281958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:09.927236845 +0000 UTC m=+569.386407832" watchObservedRunningTime="2026-11-03 17:43:09.927813331 +0000 UTC m=+569.386984306"
Nov 03 17:43:09.978403 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:09.978367    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hbbwk"]
Nov 03 17:43:09.978638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:09.978602    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk" podUID="3b6311df-8744-4121-b919-2df5aba30d0e" containerName="authorino" containerID="cri-o://c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02" gracePeriod=30
Nov 03 17:43:10.230766 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.230735    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk"
Nov 03 17:43:10.383317 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.383243    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6311df-8744-4121-b919-2df5aba30d0e-kube-api-access-tjk6f\" (UniqueName: \"kubernetes.io/projected/3b6311df-8744-4121-b919-2df5aba30d0e-kube-api-access-tjk6f\") pod \"3b6311df-8744-4121-b919-2df5aba30d0e\" (UID: \"3b6311df-8744-4121-b919-2df5aba30d0e\") "
Nov 03 17:43:10.385457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.385400    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6311df-8744-4121-b919-2df5aba30d0e-kube-api-access-tjk6f" pod "3b6311df-8744-4121-b919-2df5aba30d0e" (UID: "3b6311df-8744-4121-b919-2df5aba30d0e"). InnerVolumeSpecName "kube-api-access-tjk6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:43:10.484308 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.484270    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjk6f\" (UniqueName: \"kubernetes.io/projected/3b6311df-8744-4121-b919-2df5aba30d0e-kube-api-access-tjk6f\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:43:10.710672 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.710606    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05ba4b7-0164-4e6a-8cdd-b3add04a181f" path="/var/lib/kubelet/pods/b05ba4b7-0164-4e6a-8cdd-b3add04a181f/volumes"
Nov 03 17:43:10.812708 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.812666    4493 generic.go:356] "Generic (PLEG): container finished" podID="3b6311df-8744-4121-b919-2df5aba30d0e" containerID="c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02" exitCode=0
Nov 03 17:43:10.812708 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.812705    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk"
Nov 03 17:43:10.813323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.812748    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk" event={"ID":"3b6311df-8744-4121-b919-2df5aba30d0e","Type":"ContainerDied","Data":"c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02"}
Nov 03 17:43:10.813323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.812793    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-hbbwk" event={"ID":"3b6311df-8744-4121-b919-2df5aba30d0e","Type":"ContainerDied","Data":"d468ae7b8c2f3a4fac41e52ed7ad4d870edf317eb2e5d977958d6632719bee2f"}
Nov 03 17:43:10.813323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.812817    4493 scope.go:122] "RemoveContainer" containerID="c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02"
Nov 03 17:43:10.821095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.821077    4493 scope.go:122] "RemoveContainer" containerID="c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02"
Nov 03 17:43:10.821381 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:43:10.821359    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02\": container with ID starting with c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02 not found: ID does not exist" containerID="c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02"
Nov 03 17:43:10.821461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.821391    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02"} err="failed to get container status \"c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02\": rpc error: code = NotFound desc = could not find container \"c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02\": container with ID starting with c2c1c227ccc6fade2e5722e003eda06413f320301d3c65814953c6c95a964b02 not found: ID does not exist"
Nov 03 17:43:10.981326 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.981293    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hbbwk"]
Nov 03 17:43:10.987273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:10.987242    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-hbbwk"]
Nov 03 17:43:12.710852 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:12.710822    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6311df-8744-4121-b919-2df5aba30d0e" path="/var/lib/kubelet/pods/3b6311df-8744-4121-b919-2df5aba30d0e/volumes"
Nov 03 17:43:36.991185 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:43:36.991146    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:43:51.402660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.402611    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"]
Nov 03 17:43:51.403043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.402927    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3b6311df-8744-4121-b919-2df5aba30d0e" containerName="authorino"
Nov 03 17:43:51.403043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.402940    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6311df-8744-4121-b919-2df5aba30d0e" containerName="authorino"
Nov 03 17:43:51.403043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.403013    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6311df-8744-4121-b919-2df5aba30d0e" containerName="authorino"
Nov 03 17:43:51.405386 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.405371    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:51.467716 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.467685    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"]
Nov 03 17:43:51.518623 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.518578    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfj9v\" (UniqueName: \"kubernetes.io/projected/ab184125-d586-48cc-944c-8ec77fb048e0-kube-api-access-zfj9v\") pod \"llmisvc-controller-manager-6466c554d8-dbx77\" (UID: \"ab184125-d586-48cc-944c-8ec77fb048e0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:51.518796 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.518697    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab184125-d586-48cc-944c-8ec77fb048e0-cert\") pod \"llmisvc-controller-manager-6466c554d8-dbx77\" (UID: \"ab184125-d586-48cc-944c-8ec77fb048e0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:51.619522 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.619482    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab184125-d586-48cc-944c-8ec77fb048e0-cert\") pod \"llmisvc-controller-manager-6466c554d8-dbx77\" (UID: \"ab184125-d586-48cc-944c-8ec77fb048e0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:51.619704 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.619626    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfj9v\" (UniqueName: \"kubernetes.io/projected/ab184125-d586-48cc-944c-8ec77fb048e0-kube-api-access-zfj9v\") pod \"llmisvc-controller-manager-6466c554d8-dbx77\" (UID: \"ab184125-d586-48cc-944c-8ec77fb048e0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:51.622156 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.622133    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab184125-d586-48cc-944c-8ec77fb048e0-cert\") pod \"llmisvc-controller-manager-6466c554d8-dbx77\" (UID: \"ab184125-d586-48cc-944c-8ec77fb048e0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:51.684334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.684270    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfj9v\" (UniqueName: \"kubernetes.io/projected/ab184125-d586-48cc-944c-8ec77fb048e0-kube-api-access-zfj9v\") pod \"llmisvc-controller-manager-6466c554d8-dbx77\" (UID: \"ab184125-d586-48cc-944c-8ec77fb048e0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:51.716213 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:51.716188    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:52.089808 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:52.089787    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"]
Nov 03 17:43:52.092385 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:43:52.092363    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab184125_d586_48cc_944c_8ec77fb048e0.slice/crio-f0040addc3f078dfe62b793b867dada6bc5736c4fc33a9869b263c616fd5b106 WatchSource:0}: Error finding container f0040addc3f078dfe62b793b867dada6bc5736c4fc33a9869b263c616fd5b106: Status 404 returned error can't find the container with id f0040addc3f078dfe62b793b867dada6bc5736c4fc33a9869b263c616fd5b106
Nov 03 17:43:52.984452 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:52.984421    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77" event={"ID":"ab184125-d586-48cc-944c-8ec77fb048e0","Type":"ContainerStarted","Data":"0e217073cd63cde3e13ffa8c701d451b904e4f3f405b38ea0b3cba221c85b28d"}
Nov 03 17:43:52.984844 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:52.984467    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77" event={"ID":"ab184125-d586-48cc-944c-8ec77fb048e0","Type":"ContainerStarted","Data":"f0040addc3f078dfe62b793b867dada6bc5736c4fc33a9869b263c616fd5b106"}
Nov 03 17:43:52.984844 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:52.984568    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:43:53.106919 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:43:53.106795    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77" podStartSLOduration=1.401155242 podStartE2EDuration="2.106781811s" totalImagesPullingTime="705.626569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:53.106173651 +0000 UTC m=+612.565344638" watchObservedRunningTime="2026-11-03 17:43:53.106781811 +0000 UTC m=+612.565952826"
Nov 03 17:44:06.990847 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:44:06.990810    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:44:23.990771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:23.990740    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-dbx77"
Nov 03 17:44:24.165638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.165597    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"]
Nov 03 17:44:24.165824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.165800    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5" podUID="3e3fbe0d-a31f-420a-b172-fef6fef276d7" containerName="manager" containerID="cri-o://1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc" gracePeriod=30
Nov 03 17:44:24.403745 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.403723    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:44:24.488165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.488133    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e3fbe0d-a31f-420a-b172-fef6fef276d7-cert\" (UniqueName: \"kubernetes.io/secret/3e3fbe0d-a31f-420a-b172-fef6fef276d7-cert\") pod \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") "
Nov 03 17:44:24.488309 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.488208    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3fbe0d-a31f-420a-b172-fef6fef276d7-kube-api-access-wk6rc\" (UniqueName: \"kubernetes.io/projected/3e3fbe0d-a31f-420a-b172-fef6fef276d7-kube-api-access-wk6rc\") pod \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\" (UID: \"3e3fbe0d-a31f-420a-b172-fef6fef276d7\") "
Nov 03 17:44:24.490458 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.490432    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3fbe0d-a31f-420a-b172-fef6fef276d7-kube-api-access-wk6rc" pod "3e3fbe0d-a31f-420a-b172-fef6fef276d7" (UID: "3e3fbe0d-a31f-420a-b172-fef6fef276d7"). InnerVolumeSpecName "kube-api-access-wk6rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:44:24.490458 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.490440    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3fbe0d-a31f-420a-b172-fef6fef276d7-cert" pod "3e3fbe0d-a31f-420a-b172-fef6fef276d7" (UID: "3e3fbe0d-a31f-420a-b172-fef6fef276d7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:44:24.589583 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.589487    4493 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e3fbe0d-a31f-420a-b172-fef6fef276d7-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:44:24.589583 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:24.589533    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6rc\" (UniqueName: \"kubernetes.io/projected/3e3fbe0d-a31f-420a-b172-fef6fef276d7-kube-api-access-wk6rc\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:44:25.111203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.111170    4493 generic.go:356] "Generic (PLEG): container finished" podID="3e3fbe0d-a31f-420a-b172-fef6fef276d7" containerID="1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc" exitCode=0
Nov 03 17:44:25.111699 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.111233    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5" event={"ID":"3e3fbe0d-a31f-420a-b172-fef6fef276d7","Type":"ContainerDied","Data":"1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc"}
Nov 03 17:44:25.111699 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.111241    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"
Nov 03 17:44:25.111699 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.111279    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5" event={"ID":"3e3fbe0d-a31f-420a-b172-fef6fef276d7","Type":"ContainerDied","Data":"9a96d774d890674deb8015f72910a08c33dfd2e8b9aa87fc799bd103807d52db"}
Nov 03 17:44:25.111699 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.111300    4493 scope.go:122] "RemoveContainer" containerID="1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc"
Nov 03 17:44:25.119660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.119640    4493 scope.go:122] "RemoveContainer" containerID="1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc"
Nov 03 17:44:25.119983 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:44:25.119963    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc\": container with ID starting with 1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc not found: ID does not exist" containerID="1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc"
Nov 03 17:44:25.120054 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.119992    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc"} err="failed to get container status \"1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc\": rpc error: code = NotFound desc = could not find container \"1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc\": container with ID starting with 1dafb47f106b09251467c2b836f46a814d78e9268da2fa6c38c878d22fa3dacc not found: ID does not exist"
Nov 03 17:44:25.281487 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.281438    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"]
Nov 03 17:44:25.332011 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:25.331978    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-zq5x5"]
Nov 03 17:44:26.710369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:44:26.710335    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3fbe0d-a31f-420a-b172-fef6fef276d7" path="/var/lib/kubelet/pods/3e3fbe0d-a31f-420a-b172-fef6fef276d7/volumes"
Nov 03 17:44:36.990485 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:44:36.990450    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:45:06.991767 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:45:06.991689    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:45:18.245008 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.244962    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"]
Nov 03 17:45:18.245639 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.245406    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3e3fbe0d-a31f-420a-b172-fef6fef276d7" containerName="manager"
Nov 03 17:45:18.245639 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.245425    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3fbe0d-a31f-420a-b172-fef6fef276d7" containerName="manager"
Nov 03 17:45:18.245639 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.245568    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3fbe0d-a31f-420a-b172-fef6fef276d7" containerName="manager"
Nov 03 17:45:18.247488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.247468    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.317406 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.317370    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 17:45:18.323198 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323174    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323321 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323208    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323321 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323242    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323381    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323464    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323617 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323499    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323695 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323633    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j782z\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-kube-api-access-j782z\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323695 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323675    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.323798 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.323701    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.360072 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.360037    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"]
Nov 03 17:45:18.394239 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.394203    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqhn9\""
Nov 03 17:45:18.424132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424098    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424135    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424169    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424198    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424237    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424261    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424327    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j782z\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-kube-api-access-j782z\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424358    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424381    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424652    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424727    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424983 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424951    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.424983 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.424968    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.425346 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.425325    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.426857 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.426836    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.427086 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.427070    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.489003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.488969    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.489167 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.489078    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j782z\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-kube-api-access-j782z\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rww5d\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.561705 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.561611    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:18.994241 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:18.994207    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"]
Nov 03 17:45:18.997158 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:45:18.997123    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf715a9_6cd9_4a37_844b_ee1356f752f2.slice/crio-cbdec06a071d4fd239918edf1b3f316d0a72eb4af038c9257234c55fdf2fa5f7 WatchSource:0}: Error finding container cbdec06a071d4fd239918edf1b3f316d0a72eb4af038c9257234c55fdf2fa5f7: Status 404 returned error can't find the container with id cbdec06a071d4fd239918edf1b3f316d0a72eb4af038c9257234c55fdf2fa5f7
Nov 03 17:45:19.330834 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:19.330744    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" event={"ID":"ecf715a9-6cd9-4a37-844b-ee1356f752f2","Type":"ContainerStarted","Data":"cbdec06a071d4fd239918edf1b3f316d0a72eb4af038c9257234c55fdf2fa5f7"}
Nov 03 17:45:22.377911 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:22.377857    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:22.378194 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:22.377946    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:22.378194 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:22.377975    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:23.349611 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:23.349575    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" event={"ID":"ecf715a9-6cd9-4a37-844b-ee1356f752f2","Type":"ContainerStarted","Data":"02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1"}
Nov 03 17:45:23.536900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:23.528934    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podStartSLOduration=2.150936331 podStartE2EDuration="5.528915782s" totalImagesPullingTime="3.377979451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:23.528349474 +0000 UTC m=+702.987520460" watchObservedRunningTime="2026-11-03 17:45:23.528915782 +0000 UTC m=+702.988086770"
Nov 03 17:45:23.562100 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:23.562055    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:23.563515 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:23.563490    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:45:23.563640 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:23.563533    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:45:24.562394 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:24.562358    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:45:24.562813 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:24.562423    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:45:25.562323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:25.562277    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:45:25.562506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:25.562337    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:45:26.562733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:26.562687    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:45:26.563252 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:26.562741    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:45:27.562215 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:27.562181    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:45:27.562385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:27.562246    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:45:28.562197 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:28.562152    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Nov 03 17:45:28.562724 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:28.562205    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Nov 03 17:45:30.563235 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:30.563190    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:45:30.563738 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:30.563269    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:45:31.605029 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:31.604996    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:45:31.605405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:31.605065    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:45:32.037383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:32.037339    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:32.037705 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:32.037653    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:32.038334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:32.038316    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 17:45:36.990346 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:45:36.990309    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:45:44.451330 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.451298    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b76779445-xg76x"]
Nov 03 17:45:44.451822 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.451485    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b76779445-xg76x" podUID="654707e4-2300-419f-9d26-151ea79567ab" containerName="authorino" containerID="cri-o://c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21" gracePeriod=30
Nov 03 17:45:44.697094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.697069    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:45:44.758450 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.758425    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/654707e4-2300-419f-9d26-151ea79567ab-tls-cert\" (UniqueName: \"kubernetes.io/secret/654707e4-2300-419f-9d26-151ea79567ab-tls-cert\") pod \"654707e4-2300-419f-9d26-151ea79567ab\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") "
Nov 03 17:45:44.758594 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.758553    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654707e4-2300-419f-9d26-151ea79567ab-kube-api-access-rjd7s\" (UniqueName: \"kubernetes.io/projected/654707e4-2300-419f-9d26-151ea79567ab-kube-api-access-rjd7s\") pod \"654707e4-2300-419f-9d26-151ea79567ab\" (UID: \"654707e4-2300-419f-9d26-151ea79567ab\") "
Nov 03 17:45:44.760558 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.760534    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654707e4-2300-419f-9d26-151ea79567ab-kube-api-access-rjd7s" pod "654707e4-2300-419f-9d26-151ea79567ab" (UID: "654707e4-2300-419f-9d26-151ea79567ab"). InnerVolumeSpecName "kube-api-access-rjd7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:45:44.767801 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.767771    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654707e4-2300-419f-9d26-151ea79567ab-tls-cert" pod "654707e4-2300-419f-9d26-151ea79567ab" (UID: "654707e4-2300-419f-9d26-151ea79567ab"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:45:44.859310 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.859276    4493 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/654707e4-2300-419f-9d26-151ea79567ab-tls-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:45:44.859310 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:44.859304    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjd7s\" (UniqueName: \"kubernetes.io/projected/654707e4-2300-419f-9d26-151ea79567ab-kube-api-access-rjd7s\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:45:45.438763 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.438722    4493 generic.go:356] "Generic (PLEG): container finished" podID="654707e4-2300-419f-9d26-151ea79567ab" containerID="c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21" exitCode=0
Nov 03 17:45:45.438763 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.438765    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b76779445-xg76x"
Nov 03 17:45:45.439033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.438803    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b76779445-xg76x" event={"ID":"654707e4-2300-419f-9d26-151ea79567ab","Type":"ContainerDied","Data":"c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21"}
Nov 03 17:45:45.439033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.438844    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b76779445-xg76x" event={"ID":"654707e4-2300-419f-9d26-151ea79567ab","Type":"ContainerDied","Data":"66005c85d720a0b908d027101b9cc353ffa554f34e350e89d17e9cf0978dea5b"}
Nov 03 17:45:45.439033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.438860    4493 scope.go:122] "RemoveContainer" containerID="c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21"
Nov 03 17:45:45.447258 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.447242    4493 scope.go:122] "RemoveContainer" containerID="c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21"
Nov 03 17:45:45.447535 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:45:45.447517    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21\": container with ID starting with c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21 not found: ID does not exist" containerID="c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21"
Nov 03 17:45:45.447611 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.447544    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21"} err="failed to get container status \"c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21\": rpc error: code = NotFound desc = could not find container \"c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21\": container with ID starting with c74305b96c715ac5e348cfa33bc3f77166d39306ac51240cc104839cbe5b2b21 not found: ID does not exist"
Nov 03 17:45:45.608895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.608842    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b76779445-xg76x"]
Nov 03 17:45:45.612102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:45.612075    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b76779445-xg76x"]
Nov 03 17:45:46.710442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:46.710405    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654707e4-2300-419f-9d26-151ea79567ab" path="/var/lib/kubelet/pods/654707e4-2300-419f-9d26-151ea79567ab/volumes"
Nov 03 17:45:48.524379 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.524352    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"]
Nov 03 17:45:48.524781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.524632    4493 cpu_manager.go:414] "containerMap: removing container" podUID="654707e4-2300-419f-9d26-151ea79567ab" containerName="authorino"
Nov 03 17:45:48.524781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.524642    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654707e4-2300-419f-9d26-151ea79567ab" containerName="authorino"
Nov 03 17:45:48.524781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.524712    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="654707e4-2300-419f-9d26-151ea79567ab" containerName="authorino"
Nov 03 17:45:48.529236 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.529220    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.592803 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.592754    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc0e29-5de3-43f6-a09a-e46774183e46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.592980 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.592862    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.592980 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.592916    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqp2\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-kube-api-access-sgqp2\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.592980 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.592970    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.593115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.593049    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.593115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.593069    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.593115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.593088    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.593115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.593107    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.593298 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.593127    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.694688 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.694643    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.694890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.694694    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.694890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.694716    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.694890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.694742    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.694890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.694770    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.694924    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.694971    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc0e29-5de3-43f6-a09a-e46774183e46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.695015    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.695035    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqp2\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-kube-api-access-sgqp2\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.695114    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.695141    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.695201    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695483 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.695462    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.695690 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.695639    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc0e29-5de3-43f6-a09a-e46774183e46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.697273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.697249    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.697500 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.697482    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.746189 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.746163    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"]
Nov 03 17:45:48.758974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.758947    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.759302 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.759276    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqp2\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-kube-api-access-sgqp2\") pod \"maas-default-gateway-openshift-default-5df48fcc99-29cqb\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:48.847387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:48.847318    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:49.279461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.279441    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"]
Nov 03 17:45:49.282090 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:45:49.282064    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dc0e29_5de3_43f6_a09a_e46774183e46.slice/crio-55528ce7674565406121d8cac384f1a80158adb66234b7041af414046039be62 WatchSource:0}: Error finding container 55528ce7674565406121d8cac384f1a80158adb66234b7041af414046039be62: Status 404 returned error can't find the container with id 55528ce7674565406121d8cac384f1a80158adb66234b7041af414046039be62
Nov 03 17:45:49.284121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.284084    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:49.284205 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.284152    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:49.284205 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.284183    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:45:49.458405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.458366    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" event={"ID":"93dc0e29-5de3-43f6-a09a-e46774183e46","Type":"ContainerStarted","Data":"4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8"}
Nov 03 17:45:49.458607 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.458424    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" event={"ID":"93dc0e29-5de3-43f6-a09a-e46774183e46","Type":"ContainerStarted","Data":"55528ce7674565406121d8cac384f1a80158adb66234b7041af414046039be62"}
Nov 03 17:45:49.590240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.590137    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podStartSLOduration=1.590122333 podStartE2EDuration="1.590122333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:49.589701828 +0000 UTC m=+729.048872815" watchObservedRunningTime="2026-11-03 17:45:49.590122333 +0000 UTC m=+729.049293394"
Nov 03 17:45:49.848359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:49.848269    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:50.849811 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:50.849776    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-29cqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:45:50.850248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:50.849829    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:45:51.848187 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:51.848151    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-29cqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:45:51.848361 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:51.848203    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:45:52.848470 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:52.848436    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-29cqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:45:52.848897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:52.848490    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:45:53.848050 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:53.848016    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-29cqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:45:53.848221 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:53.848065    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:45:54.847889 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:54.847841    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-29cqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:45:54.848296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:54.847944    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:45:55.848434 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:55.848394    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-29cqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Nov 03 17:45:55.848960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:55.848457    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Nov 03 17:45:57.359629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:57.359590    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:57.360092 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:57.360044    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:45:57.360969 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:45:57.360948    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:46:06.990662 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:46:06.990627    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:46:36.991753 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:46:36.991708    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:47:06.319555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.319520    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"]
Nov 03 17:47:06.322770 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.322753    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.358399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.358371    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhtk\" (UniqueName: \"kubernetes.io/projected/0a38ae74-1b59-4b19-a578-1ebc71bd368e-kube-api-access-6vhtk\") pod \"payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.358514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.358428    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a38ae74-1b59-4b19-a578-1ebc71bd368e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.377598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.377575    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-776ba7\""
Nov 03 17:47:06.427511 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.427478    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-776ba7-dockercfg-8vpds\""
Nov 03 17:47:06.428737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.428718    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"]
Nov 03 17:47:06.459269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.459240    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhtk\" (UniqueName: \"kubernetes.io/projected/0a38ae74-1b59-4b19-a578-1ebc71bd368e-kube-api-access-6vhtk\") pod \"payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.459407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.459297    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a38ae74-1b59-4b19-a578-1ebc71bd368e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.459967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.459949    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a38ae74-1b59-4b19-a578-1ebc71bd368e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.480225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.480201    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"]
Nov 03 17:47:06.483399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.483383    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.523633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.523594    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhtk\" (UniqueName: \"kubernetes.io/projected/0a38ae74-1b59-4b19-a578-1ebc71bd368e-kube-api-access-6vhtk\") pod \"payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.533007 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.532968    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"]
Nov 03 17:47:06.560745 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.560712    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264bf7be-e144-4ade-b8d2-e648ee3ba04f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.560915 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.560859    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpsnp\" (UniqueName: \"kubernetes.io/projected/264bf7be-e144-4ade-b8d2-e648ee3ba04f-kube-api-access-cpsnp\") pod \"payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.635223 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.635147    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:06.661612 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.661585    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264bf7be-e144-4ade-b8d2-e648ee3ba04f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.661728 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.661669    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsnp\" (UniqueName: \"kubernetes.io/projected/264bf7be-e144-4ade-b8d2-e648ee3ba04f-kube-api-access-cpsnp\") pod \"payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.662616 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.662598    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264bf7be-e144-4ade-b8d2-e648ee3ba04f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.726947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.726908    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpsnp\" (UniqueName: \"kubernetes.io/projected/264bf7be-e144-4ade-b8d2-e648ee3ba04f-kube-api-access-cpsnp\") pod \"payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.795373 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:06.795348    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:06.990497 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:47:06.990452    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:47:07.009328 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:07.009300    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"]
Nov 03 17:47:07.011796 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:47:07.011773    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a38ae74_1b59_4b19_a578_1ebc71bd368e.slice/crio-e31077d5aba2736cf116a91a59f50014bff646c19256cc17ddfe29293b0191f3 WatchSource:0}: Error finding container e31077d5aba2736cf116a91a59f50014bff646c19256cc17ddfe29293b0191f3: Status 404 returned error can't find the container with id e31077d5aba2736cf116a91a59f50014bff646c19256cc17ddfe29293b0191f3
Nov 03 17:47:07.013570 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:07.013556    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 17:47:07.168864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:07.168765    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"]
Nov 03 17:47:07.172447 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:47:07.172412    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264bf7be_e144_4ade_b8d2_e648ee3ba04f.slice/crio-55b954e4a5bcd916c0878284eb06591cb72046cbb6c21e0ab6e3f662ae1597fd WatchSource:0}: Error finding container 55b954e4a5bcd916c0878284eb06591cb72046cbb6c21e0ab6e3f662ae1597fd: Status 404 returned error can't find the container with id 55b954e4a5bcd916c0878284eb06591cb72046cbb6c21e0ab6e3f662ae1597fd
Nov 03 17:47:07.765001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:07.764966    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" event={"ID":"0a38ae74-1b59-4b19-a578-1ebc71bd368e","Type":"ContainerStarted","Data":"e31077d5aba2736cf116a91a59f50014bff646c19256cc17ddfe29293b0191f3"}
Nov 03 17:47:07.766062 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:07.766042    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" event={"ID":"264bf7be-e144-4ade-b8d2-e648ee3ba04f","Type":"ContainerStarted","Data":"55b954e4a5bcd916c0878284eb06591cb72046cbb6c21e0ab6e3f662ae1597fd"}
Nov 03 17:47:09.025856 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.025825    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"]
Nov 03 17:47:09.034426 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.034368    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.087025 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.086986    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cc8ddf-6721-428b-972c-68a77603096a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.087217 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.087046    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rqt\" (UniqueName: \"kubernetes.io/projected/56cc8ddf-6721-428b-972c-68a77603096a-kube-api-access-c5rqt\") pod \"payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.090240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.090201    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"]
Nov 03 17:47:09.139033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.138988    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-42c162\""
Nov 03 17:47:09.139033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.139017    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-42c162-dockercfg-k47lh\""
Nov 03 17:47:09.187941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.187857    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cc8ddf-6721-428b-972c-68a77603096a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.188121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.187964    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rqt\" (UniqueName: \"kubernetes.io/projected/56cc8ddf-6721-428b-972c-68a77603096a-kube-api-access-c5rqt\") pod \"payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.188633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.188607    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cc8ddf-6721-428b-972c-68a77603096a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.264782 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.264748    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rqt\" (UniqueName: \"kubernetes.io/projected/56cc8ddf-6721-428b-972c-68a77603096a-kube-api-access-c5rqt\") pod \"payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.356064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.355981    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:09.760105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.760075    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"]
Nov 03 17:47:09.761910 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:47:09.761855    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cc8ddf_6721_428b_972c_68a77603096a.slice/crio-dea1e3c135494d88cbed18c1f10b16383c47e349c7236e1fdb96a14e6f7a658b WatchSource:0}: Error finding container dea1e3c135494d88cbed18c1f10b16383c47e349c7236e1fdb96a14e6f7a658b: Status 404 returned error can't find the container with id dea1e3c135494d88cbed18c1f10b16383c47e349c7236e1fdb96a14e6f7a658b
Nov 03 17:47:09.779248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:09.779210    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf" event={"ID":"56cc8ddf-6721-428b-972c-68a77603096a","Type":"ContainerStarted","Data":"dea1e3c135494d88cbed18c1f10b16383c47e349c7236e1fdb96a14e6f7a658b"}
Nov 03 17:47:10.786144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:10.786111    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" event={"ID":"264bf7be-e144-4ade-b8d2-e648ee3ba04f","Type":"ContainerStarted","Data":"2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c"}
Nov 03 17:47:10.786636 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:10.786219    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:10.787665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:10.787638    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf" event={"ID":"56cc8ddf-6721-428b-972c-68a77603096a","Type":"ContainerStarted","Data":"6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2"}
Nov 03 17:47:10.787792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:10.787695    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:10.788988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:10.788969    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" event={"ID":"0a38ae74-1b59-4b19-a578-1ebc71bd368e","Type":"ContainerStarted","Data":"8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83"}
Nov 03 17:47:10.789110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:10.789099    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:10.906867 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:10.906760    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" podStartSLOduration=2.135284441 podStartE2EDuration="4.906745625s" totalImagesPullingTime="2.771461184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:10.906519356 +0000 UTC m=+810.365690335" watchObservedRunningTime="2026-11-03 17:47:10.906745625 +0000 UTC m=+810.365916619"
Nov 03 17:47:11.028648 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:11.028579    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf" podStartSLOduration=2.225675384 podStartE2EDuration="3.028560568s" totalImagesPullingTime="802.885184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:11.027370341 +0000 UTC m=+810.486541327" watchObservedRunningTime="2026-11-03 17:47:11.028560568 +0000 UTC m=+810.487731554"
Nov 03 17:47:16.572424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.572368    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" podStartSLOduration=7.702799108 podStartE2EDuration="10.572353219s" totalImagesPullingTime="2.869554111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:11.150070934 +0000 UTC m=+810.609241919" watchObservedRunningTime="2026-11-03 17:47:16.572353219 +0000 UTC m=+816.031524203"
Nov 03 17:47:16.572896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.572496    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"]
Nov 03 17:47:16.576650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.576627    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:16.635921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.635894    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"]
Nov 03 17:47:16.661363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.661312    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqkf\" (UniqueName: \"kubernetes.io/projected/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-kube-api-access-qcqkf\") pod \"payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:16.661582 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.661518    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:16.681164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.681142    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-548ca1-dockercfg-7gknk\""
Nov 03 17:47:16.681164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.681150    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-548ca1\""
Nov 03 17:47:16.762017 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.761985    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqkf\" (UniqueName: \"kubernetes.io/projected/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-kube-api-access-qcqkf\") pod \"payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:16.762162 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.762042    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:16.762714 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.762693    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:16.826054 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.825988    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqkf\" (UniqueName: \"kubernetes.io/projected/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-kube-api-access-qcqkf\") pod \"payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:16.891404 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:16.891372    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:17.264280 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:17.264259    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"]
Nov 03 17:47:17.266600 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:47:17.266572    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0e42b5_2d61_4ddc_b1e8_2230abf994c7.slice/crio-075afb8a7ca8cf101c98e64d76a3b1c5136430b73c8c24b583e3810c6888e3e4 WatchSource:0}: Error finding container 075afb8a7ca8cf101c98e64d76a3b1c5136430b73c8c24b583e3810c6888e3e4: Status 404 returned error can't find the container with id 075afb8a7ca8cf101c98e64d76a3b1c5136430b73c8c24b583e3810c6888e3e4
Nov 03 17:47:17.818131 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:17.818094    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" event={"ID":"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7","Type":"ContainerStarted","Data":"3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa"}
Nov 03 17:47:17.818555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:17.818153    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" event={"ID":"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7","Type":"ContainerStarted","Data":"075afb8a7ca8cf101c98e64d76a3b1c5136430b73c8c24b583e3810c6888e3e4"}
Nov 03 17:47:17.818555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:17.818244    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:17.941368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:17.941315    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" podStartSLOduration=1.9413015150000001 podStartE2EDuration="1.941301515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:17.940702421 +0000 UTC m=+817.399873411" watchObservedRunningTime="2026-11-03 17:47:17.941301515 +0000 UTC m=+817.400472500"
Nov 03 17:47:21.641169 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.641133    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"]
Nov 03 17:47:21.644630 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.644612    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.701669 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.701642    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7786ff62-727a-4d61-98f6-e3f1071c97f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.701793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.701680    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7kr\" (UniqueName: \"kubernetes.io/projected/7786ff62-727a-4d61-98f6-e3f1071c97f7-kube-api-access-qm7kr\") pod \"payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.706006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.705978    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"]
Nov 03 17:47:21.754850 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.754813    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-73a606-dockercfg-cplk6\""
Nov 03 17:47:21.754850 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.754828    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-73a606\""
Nov 03 17:47:21.756239 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.756221    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"]
Nov 03 17:47:21.759573 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.759559    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:21.795925 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.795902    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 17:47:21.796059 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.795997    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 17:47:21.796059 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.796041    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 17:47:21.802537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.802509    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7kr\" (UniqueName: \"kubernetes.io/projected/7786ff62-727a-4d61-98f6-e3f1071c97f7-kube-api-access-qm7kr\") pod \"payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.802681 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.802555    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2fea08-041f-41d9-95be-811f63cc4434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:21.802681 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.802625    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7ft\" (UniqueName: \"kubernetes.io/projected/3b2fea08-041f-41d9-95be-811f63cc4434-kube-api-access-5k7ft\") pod \"payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:21.802818 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.802800    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7786ff62-727a-4d61-98f6-e3f1071c97f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.803391 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.803368    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7786ff62-727a-4d61-98f6-e3f1071c97f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.814579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.814553    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"]
Nov 03 17:47:21.868809 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.868778    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7kr\" (UniqueName: \"kubernetes.io/projected/7786ff62-727a-4d61-98f6-e3f1071c97f7-kube-api-access-qm7kr\") pod \"payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.903777 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.903708    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2fea08-041f-41d9-95be-811f63cc4434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:21.903943 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.903828    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7ft\" (UniqueName: \"kubernetes.io/projected/3b2fea08-041f-41d9-95be-811f63cc4434-kube-api-access-5k7ft\") pod \"payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:21.904451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.904435    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2fea08-041f-41d9-95be-811f63cc4434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:21.956739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.956709    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:21.967612 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:21.967584    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7ft\" (UniqueName: \"kubernetes.io/projected/3b2fea08-041f-41d9-95be-811f63cc4434-kube-api-access-5k7ft\") pod \"payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:22.072548 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.072517    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:22.332038 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.332015    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"]
Nov 03 17:47:22.334404 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:47:22.334371    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7786ff62_727a_4d61_98f6_e3f1071c97f7.slice/crio-e48555f6d491ed431fafb689aaaf625133cf3740fdefad5cb16f514d0ce474dd WatchSource:0}: Error finding container e48555f6d491ed431fafb689aaaf625133cf3740fdefad5cb16f514d0ce474dd: Status 404 returned error can't find the container with id e48555f6d491ed431fafb689aaaf625133cf3740fdefad5cb16f514d0ce474dd
Nov 03 17:47:22.446759 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.446645    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"]
Nov 03 17:47:22.449230 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:47:22.449205    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2fea08_041f_41d9_95be_811f63cc4434.slice/crio-df7ac362983801b40579e891675fa5dbf3f4f07a521338cea98b13f52e8d10e2 WatchSource:0}: Error finding container df7ac362983801b40579e891675fa5dbf3f4f07a521338cea98b13f52e8d10e2: Status 404 returned error can't find the container with id df7ac362983801b40579e891675fa5dbf3f4f07a521338cea98b13f52e8d10e2
Nov 03 17:47:22.841179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.841138    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" event={"ID":"7786ff62-727a-4d61-98f6-e3f1071c97f7","Type":"ContainerStarted","Data":"5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479"}
Nov 03 17:47:22.841637 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.841198    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" event={"ID":"7786ff62-727a-4d61-98f6-e3f1071c97f7","Type":"ContainerStarted","Data":"e48555f6d491ed431fafb689aaaf625133cf3740fdefad5cb16f514d0ce474dd"}
Nov 03 17:47:22.841637 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.841220    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:22.842692 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.842673    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" event={"ID":"3b2fea08-041f-41d9-95be-811f63cc4434","Type":"ContainerStarted","Data":"fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3"}
Nov 03 17:47:22.842829 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.842721    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:22.842829 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.842745    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" event={"ID":"3b2fea08-041f-41d9-95be-811f63cc4434","Type":"ContainerStarted","Data":"df7ac362983801b40579e891675fa5dbf3f4f07a521338cea98b13f52e8d10e2"}
Nov 03 17:47:22.961394 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:22.961339    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" podStartSLOduration=1.961324369 podStartE2EDuration="1.961324369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:22.960822797 +0000 UTC m=+822.419993783" watchObservedRunningTime="2026-11-03 17:47:22.961324369 +0000 UTC m=+822.420495354"
Nov 03 17:47:23.081346 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:23.081290    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" podStartSLOduration=2.081277091 podStartE2EDuration="2.081277091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:23.080450888 +0000 UTC m=+822.539621885" watchObservedRunningTime="2026-11-03 17:47:23.081277091 +0000 UTC m=+822.540448075"
Nov 03 17:47:28.825889 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:28.825842    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 17:47:33.850318 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:33.850292    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 17:47:33.850721 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:33.850359    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 17:47:36.991835 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:47:36.991792    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:47:42.976444 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:42.976410    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"]
Nov 03 17:47:42.983546 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:42.983529    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.086973 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.086938    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"]
Nov 03 17:47:43.089237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.089212    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"openshift-service-ca.crt\""
Nov 03 17:47:43.089346 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.089334    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"kube-root-ca.crt\""
Nov 03 17:47:43.089453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.089434    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"simulator-w6-f7d463-kserve-self-signed-certs\""
Nov 03 17:47:43.089453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.089440    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"default-dockercfg-snqpt\""
Nov 03 17:47:43.096849 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.096827    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-home\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.096972 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.096856    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjtj\" (UniqueName: \"kubernetes.io/projected/38c08f57-c68d-421f-980d-f66efb8c53f0-kube-api-access-srjtj\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.097021 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.097008    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-model-cache\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.097070 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.097035    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-tmp-dir\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.097070 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.097060    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c08f57-c68d-421f-980d-f66efb8c53f0-tls-certs\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.097229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.097211    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-dshm\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.197962 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.197920    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-home\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198155 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.197974    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjtj\" (UniqueName: \"kubernetes.io/projected/38c08f57-c68d-421f-980d-f66efb8c53f0-kube-api-access-srjtj\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198155 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.198058    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-model-cache\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198155 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.198089    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-tmp-dir\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198155 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.198121    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c08f57-c68d-421f-980d-f66efb8c53f0-tls-certs\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.198177    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-dshm\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198433 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.198377    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-home\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198476 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.198457    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-tmp-dir\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.198687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.198663    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-model-cache\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.200522 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.200505    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-dshm\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.200698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.200680    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c08f57-c68d-421f-980d-f66efb8c53f0-tls-certs\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.261697 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.261632    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjtj\" (UniqueName: \"kubernetes.io/projected/38c08f57-c68d-421f-980d-f66efb8c53f0-kube-api-access-srjtj\") pod \"simulator-w6-f7d463-kserve-f497bc67c-q4jd9\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") " pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.295495 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.295463    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:43.675719 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.675697    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"]
Nov 03 17:47:43.678316 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:47:43.678285    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38c08f57_c68d_421f_980d_f66efb8c53f0.slice/crio-f98fe38c1a2c8d1572e9f3bdee8d7391bab53c0f5ddfc1e899a01fbd81df5014 WatchSource:0}: Error finding container f98fe38c1a2c8d1572e9f3bdee8d7391bab53c0f5ddfc1e899a01fbd81df5014: Status 404 returned error can't find the container with id f98fe38c1a2c8d1572e9f3bdee8d7391bab53c0f5ddfc1e899a01fbd81df5014
Nov 03 17:47:43.932898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:43.932812    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" event={"ID":"38c08f57-c68d-421f-980d-f66efb8c53f0","Type":"ContainerStarted","Data":"f98fe38c1a2c8d1572e9f3bdee8d7391bab53c0f5ddfc1e899a01fbd81df5014"}
Nov 03 17:47:45.942999 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:45.942950    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" event={"ID":"38c08f57-c68d-421f-980d-f66efb8c53f0","Type":"ContainerStarted","Data":"523d2fdbe8d685914c1b65f5564caa6650c8da6c678fab9ca57d8a254383e3f5"}
Nov 03 17:47:46.068744 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:46.068692    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podStartSLOduration=2.482254475 podStartE2EDuration="4.06867835s" totalImagesPullingTime="1.586423875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:46.06734138 +0000 UTC m=+845.526512390" watchObservedRunningTime="2026-11-03 17:47:46.06867835 +0000 UTC m=+845.527849334"
Nov 03 17:47:46.295802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:46.295764    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:53.296528 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:53.296441    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:53.310043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:53.310014    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:47:53.987121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:47:53.987095    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:48:06.322196 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.322162    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"]
Nov 03 17:48:06.325511 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.325494    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.395582 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.395529    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b6c25-dfeb-4b1a-85d2-794d81968f68-tls-certs\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.395733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.395638    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-home\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.395733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.395676    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-model-cache\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.395733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.395725    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-tmp-dir\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.395833 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.395753    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9d8\" (UniqueName: \"kubernetes.io/projected/725b6c25-dfeb-4b1a-85d2-794d81968f68-kube-api-access-dn9d8\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.395833 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.395814    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-dshm\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.402146 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.402127    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"kube-root-ca.crt\""
Nov 03 17:48:06.429969 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.429939    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"openshift-service-ca.crt\""
Nov 03 17:48:06.430215 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.430201    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"default-dockercfg-dv5pf\""
Nov 03 17:48:06.430265 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.430233    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"simulator-w3-776ba7-kserve-self-signed-certs\""
Nov 03 17:48:06.432951 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.432933    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"]
Nov 03 17:48:06.496429 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496397    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b6c25-dfeb-4b1a-85d2-794d81968f68-tls-certs\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.496624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496436    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-home\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.496624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496566    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-model-cache\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.496624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496601    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-tmp-dir\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.496795 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496735    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9d8\" (UniqueName: \"kubernetes.io/projected/725b6c25-dfeb-4b1a-85d2-794d81968f68-kube-api-access-dn9d8\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.496851 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496805    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-dshm\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.496936 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496862    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-home\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.496936 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496925    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-model-cache\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.497095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.496944    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-tmp-dir\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.499240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.499219    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b6c25-dfeb-4b1a-85d2-794d81968f68-tls-certs\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.499352 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.499301    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-dshm\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.560903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.560855    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9d8\" (UniqueName: \"kubernetes.io/projected/725b6c25-dfeb-4b1a-85d2-794d81968f68-kube-api-access-dn9d8\") pod \"simulator-w3-776ba7-kserve-f67d57f96-6rx6j\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") " pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.637327 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:06.637254    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:06.991784 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:48:06.991684    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:48:07.020276 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:07.020250    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"]
Nov 03 17:48:07.022169 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:48:07.022143    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod725b6c25_dfeb_4b1a_85d2_794d81968f68.slice/crio-c49da14fd04bf5ff6e5182c70afa49b31fa2032975a854368ffd160f48dccff1 WatchSource:0}: Error finding container c49da14fd04bf5ff6e5182c70afa49b31fa2032975a854368ffd160f48dccff1: Status 404 returned error can't find the container with id c49da14fd04bf5ff6e5182c70afa49b31fa2032975a854368ffd160f48dccff1
Nov 03 17:48:07.037047 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:07.037015    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" event={"ID":"725b6c25-dfeb-4b1a-85d2-794d81968f68","Type":"ContainerStarted","Data":"c49da14fd04bf5ff6e5182c70afa49b31fa2032975a854368ffd160f48dccff1"}
Nov 03 17:48:08.043491 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:08.043458    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" event={"ID":"725b6c25-dfeb-4b1a-85d2-794d81968f68","Type":"ContainerStarted","Data":"f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2"}
Nov 03 17:48:08.165950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:08.165890    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podStartSLOduration=2.165859199 podStartE2EDuration="2.165859199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:08.164250479 +0000 UTC m=+867.623421464" watchObservedRunningTime="2026-11-03 17:48:08.165859199 +0000 UTC m=+867.625030186"
Nov 03 17:48:08.638435 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:08.638403    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:13.912205 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.912174    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"]
Nov 03 17:48:13.915793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.915771    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:13.965624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.965595    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-dshm\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:13.965756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.965632    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-model-cache\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:13.965756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.965729    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-home\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:13.965830 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.965810    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tmp-dir\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:13.965867 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.965827    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ng6\" (UniqueName: \"kubernetes.io/projected/381f0f27-9512-4d8c-a529-5b8cda13dc7f-kube-api-access-z7ng6\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:13.965950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.965925    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tls-certs\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:13.977233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:13.977215    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"openshift-service-ca.crt\""
Nov 03 17:48:14.026843 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.026820    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"kube-root-ca.crt\""
Nov 03 17:48:14.027000 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.026903    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"default-dockercfg-5nxkq\""
Nov 03 17:48:14.027000 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.026960    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"simulator-w2-42c162-kserve-self-signed-certs\""
Nov 03 17:48:14.029356 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.029312    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"]
Nov 03 17:48:14.066577 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.066532    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-home\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.066778 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.066607    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tmp-dir\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.066778 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.066648    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ng6\" (UniqueName: \"kubernetes.io/projected/381f0f27-9512-4d8c-a529-5b8cda13dc7f-kube-api-access-z7ng6\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.066916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.066778    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tls-certs\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.066916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.066842    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-dshm\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.066916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.066868    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-model-cache\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.067071 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.066975    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-home\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.067071 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.067016    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tmp-dir\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.067173 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.067159    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-model-cache\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.069131 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.069106    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-dshm\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.069375 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.069355    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tls-certs\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.130748 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.130717    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ng6\" (UniqueName: \"kubernetes.io/projected/381f0f27-9512-4d8c-a529-5b8cda13dc7f-kube-api-access-z7ng6\") pod \"simulator-w2-42c162-kserve-7d6dc99555-rhmp8\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") " pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.228047 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.228012    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:14.605198 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:14.605173    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"]
Nov 03 17:48:14.607689 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:48:14.607668    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod381f0f27_9512_4d8c_a529_5b8cda13dc7f.slice/crio-b89bea2dfac4d9a9b018205193b4ccafbf69f9f4555780862cf74d90b8584c13 WatchSource:0}: Error finding container b89bea2dfac4d9a9b018205193b4ccafbf69f9f4555780862cf74d90b8584c13: Status 404 returned error can't find the container with id b89bea2dfac4d9a9b018205193b4ccafbf69f9f4555780862cf74d90b8584c13
Nov 03 17:48:15.073994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:15.073957    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" event={"ID":"381f0f27-9512-4d8c-a529-5b8cda13dc7f","Type":"ContainerStarted","Data":"e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab"}
Nov 03 17:48:15.074358 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:15.074004    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" event={"ID":"381f0f27-9512-4d8c-a529-5b8cda13dc7f","Type":"ContainerStarted","Data":"b89bea2dfac4d9a9b018205193b4ccafbf69f9f4555780862cf74d90b8584c13"}
Nov 03 17:48:15.196126 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:15.196073    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podStartSLOduration=2.196057759 podStartE2EDuration="2.196057759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:15.194478676 +0000 UTC m=+874.653649687" watchObservedRunningTime="2026-11-03 17:48:15.196057759 +0000 UTC m=+874.655228743"
Nov 03 17:48:15.228611 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:15.228584    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:16.637423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:16.637391    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:16.650143 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:16.650116    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:17.093732 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:17.093705    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:48:24.228483 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:24.228451    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:24.240825 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:24.240797    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:25.128177 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:25.128146    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:48:29.748725 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.748692    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"]
Nov 03 17:48:29.752075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.752059    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:29.856919 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.856880    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7d463\"/\"premium-w6-f7d463-kserve-self-signed-certs\""
Nov 03 17:48:29.858756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.858736    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"]
Nov 03 17:48:29.907487 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.907459    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-home\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:29.907668 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.907493    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggpt\" (UniqueName: \"kubernetes.io/projected/c4780426-c057-47be-af63-4083a7fc8973-kube-api-access-mggpt\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:29.907668 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.907523    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-model-cache\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:29.907773 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.907629    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-tmp-dir\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:29.907773 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.907752    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4780426-c057-47be-af63-4083a7fc8973-tls-certs\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:29.907839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:29.907820    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-dshm\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.008937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.008831    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-dshm\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.008937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.008919    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-home\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.009176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.008944    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggpt\" (UniqueName: \"kubernetes.io/projected/c4780426-c057-47be-af63-4083a7fc8973-kube-api-access-mggpt\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.009176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.008973    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-model-cache\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.009176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.008997    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-tmp-dir\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.009176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.009050    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4780426-c057-47be-af63-4083a7fc8973-tls-certs\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.009400 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.009380    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-home\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.009503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.009455    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-tmp-dir\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.009556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.009452    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-model-cache\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.011508 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.011476    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-dshm\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.011626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.011609    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4780426-c057-47be-af63-4083a7fc8973-tls-certs\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.073276 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.073233    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggpt\" (UniqueName: \"kubernetes.io/projected/c4780426-c057-47be-af63-4083a7fc8973-kube-api-access-mggpt\") pod \"premium-w6-f7d463-kserve-6f5c9784dd-n9zwk\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") " pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.364083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.364001    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:30.745636 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:30.745605    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"]
Nov 03 17:48:30.748836 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:48:30.748809    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4780426_c057_47be_af63_4083a7fc8973.slice/crio-94f6dd846bd7c435876535adb04ea987761f50a20d9e4681f9742fa3a7c5d52b WatchSource:0}: Error finding container 94f6dd846bd7c435876535adb04ea987761f50a20d9e4681f9742fa3a7c5d52b: Status 404 returned error can't find the container with id 94f6dd846bd7c435876535adb04ea987761f50a20d9e4681f9742fa3a7c5d52b
Nov 03 17:48:31.145546 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:31.145438    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" event={"ID":"c4780426-c057-47be-af63-4083a7fc8973","Type":"ContainerStarted","Data":"2daf9b8eb31debbfc102d0cf9be65d5329abd740835630c45a5daa8697ab62c8"}
Nov 03 17:48:31.145546 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:31.145507    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" event={"ID":"c4780426-c057-47be-af63-4083a7fc8973","Type":"ContainerStarted","Data":"94f6dd846bd7c435876535adb04ea987761f50a20d9e4681f9742fa3a7c5d52b"}
Nov 03 17:48:31.267666 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:31.267615    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podStartSLOduration=2.267602419 podStartE2EDuration="2.267602419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:31.266611848 +0000 UTC m=+890.725782860" watchObservedRunningTime="2026-11-03 17:48:31.267602419 +0000 UTC m=+890.726773420"
Nov 03 17:48:31.364833 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:31.364800    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:36.993681 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:48:36.993647    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:48:40.364972 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:40.364937    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:40.377553 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:40.377531    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:41.202541 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:41.202507    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:48:42.583600 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.583559    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"]
Nov 03 17:48:42.587320 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.587298    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.622283 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.622244    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz8wp\" (UniqueName: \"kubernetes.io/projected/b6a2d2dd-25c1-423b-924a-8fa45149987a-kube-api-access-wz8wp\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.622461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.622346    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-dshm\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.622461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.622382    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-model-cache\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.622569 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.622512    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d2dd-25c1-423b-924a-8fa45149987a-tls-certs\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.622624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.622604    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-home\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.622668 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.622643    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-tmp-dir\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.691984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.691952    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"kube-root-ca.crt\""
Nov 03 17:48:42.691984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.691969    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"openshift-service-ca.crt\""
Nov 03 17:48:42.692177 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.692066    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"default-dockercfg-hr25s\""
Nov 03 17:48:42.692306 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.692287    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"simulator-w1-c76280-kserve-self-signed-certs\""
Nov 03 17:48:42.694665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.694649    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"]
Nov 03 17:48:42.723807 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.723780    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-model-cache\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.723984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.723833    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d2dd-25c1-423b-924a-8fa45149987a-tls-certs\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.723984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.723889    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-home\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.723984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.723906    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-tmp-dir\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.723984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.723948    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8wp\" (UniqueName: \"kubernetes.io/projected/b6a2d2dd-25c1-423b-924a-8fa45149987a-kube-api-access-wz8wp\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.723984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.723976    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-dshm\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.724249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.724236    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-model-cache\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.724314 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.724265    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-home\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.724418 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.724400    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-tmp-dir\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.726384 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.726358    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-dshm\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.727039 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.727013    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d2dd-25c1-423b-924a-8fa45149987a-tls-certs\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.787940 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.787914    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8wp\" (UniqueName: \"kubernetes.io/projected/b6a2d2dd-25c1-423b-924a-8fa45149987a-kube-api-access-wz8wp\") pod \"simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") " pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:42.900370 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:42.900265    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:43.286690 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:43.286654    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"]
Nov 03 17:48:43.289564 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:48:43.289532    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a2d2dd_25c1_423b_924a_8fa45149987a.slice/crio-67aa111d178d411a1a0372955c7db46a620a045697d3acf57e422c4e28212291 WatchSource:0}: Error finding container 67aa111d178d411a1a0372955c7db46a620a045697d3acf57e422c4e28212291: Status 404 returned error can't find the container with id 67aa111d178d411a1a0372955c7db46a620a045697d3acf57e422c4e28212291
Nov 03 17:48:44.207638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:44.207607    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" event={"ID":"b6a2d2dd-25c1-423b-924a-8fa45149987a","Type":"ContainerStarted","Data":"3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93"}
Nov 03 17:48:44.208053 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:44.207646    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" event={"ID":"b6a2d2dd-25c1-423b-924a-8fa45149987a","Type":"ContainerStarted","Data":"67aa111d178d411a1a0372955c7db46a620a045697d3acf57e422c4e28212291"}
Nov 03 17:48:44.329846 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:44.329792    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podStartSLOduration=2.329774638 podStartE2EDuration="2.329774638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:44.328711667 +0000 UTC m=+903.787882652" watchObservedRunningTime="2026-11-03 17:48:44.329774638 +0000 UTC m=+903.788945623"
Nov 03 17:48:44.900944 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:44.900906    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:52.900854 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:52.900815    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:52.913125 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:52.913100    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:53.260594 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:53.260568    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:48:54.204927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.204896    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"]
Nov 03 17:48:54.208568 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.208552    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.241257 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.241229    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-model-cache\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.241418 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.241265    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfv5t\" (UniqueName: \"kubernetes.io/projected/63387678-dbfd-43c5-87a1-ef0b581346af-kube-api-access-sfv5t\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.241418 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.241358    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-dshm\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.241418 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.241411    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63387678-dbfd-43c5-87a1-ef0b581346af-tls-certs\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.241577 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.241560    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-tmp-dir\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.241614 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.241591    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-home\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.314444 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.314405    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"kube-root-ca.crt\""
Nov 03 17:48:54.314630 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.314487    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"openshift-service-ca.crt\""
Nov 03 17:48:54.314630 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.314529    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"simulator-w0-07f297-kserve-self-signed-certs\""
Nov 03 17:48:54.314630 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.314566    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"default-dockercfg-dn566\""
Nov 03 17:48:54.316233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.316214    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"]
Nov 03 17:48:54.342221 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342197    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-dshm\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.342350 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342231    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63387678-dbfd-43c5-87a1-ef0b581346af-tls-certs\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.342350 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342295    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-tmp-dir\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.342350 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342323    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-home\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.342507 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342377    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-model-cache\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.342507 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342410    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfv5t\" (UniqueName: \"kubernetes.io/projected/63387678-dbfd-43c5-87a1-ef0b581346af-kube-api-access-sfv5t\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.342761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342733    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-tmp-dir\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.342904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342791    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-model-cache\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.343127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.342919    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-home\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.344646 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.344622    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-dshm\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.344927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.344909    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63387678-dbfd-43c5-87a1-ef0b581346af-tls-certs\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.406771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.406740    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfv5t\" (UniqueName: \"kubernetes.io/projected/63387678-dbfd-43c5-87a1-ef0b581346af-kube-api-access-sfv5t\") pod \"simulator-w0-07f297-kserve-589ff58c44-d5r47\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") " pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.520738 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.520700    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:48:54.904514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:54.904489    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"]
Nov 03 17:48:54.906070 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:48:54.906040    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63387678_dbfd_43c5_87a1_ef0b581346af.slice/crio-fe04278ec94ab0e3e37493c853e1606fddbf46290805983c89611bac29273fa4 WatchSource:0}: Error finding container fe04278ec94ab0e3e37493c853e1606fddbf46290805983c89611bac29273fa4: Status 404 returned error can't find the container with id fe04278ec94ab0e3e37493c853e1606fddbf46290805983c89611bac29273fa4
Nov 03 17:48:55.268487 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:55.268457    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" event={"ID":"63387678-dbfd-43c5-87a1-ef0b581346af","Type":"ContainerStarted","Data":"f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995"}
Nov 03 17:48:55.268866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:55.268497    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" event={"ID":"63387678-dbfd-43c5-87a1-ef0b581346af","Type":"ContainerStarted","Data":"fe04278ec94ab0e3e37493c853e1606fddbf46290805983c89611bac29273fa4"}
Nov 03 17:48:55.390452 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:55.390406    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podStartSLOduration=1.390393133 podStartE2EDuration="1.390393133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:55.389193911 +0000 UTC m=+914.848364895" watchObservedRunningTime="2026-11-03 17:48:55.390393133 +0000 UTC m=+914.849564154"
Nov 03 17:48:55.520861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:48:55.520779    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:49:04.521675 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:04.521643    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:49:04.534276 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:04.534255    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:49:05.322772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:05.322748    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:49:06.728160 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.728120    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"]
Nov 03 17:49:06.731966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.731943    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.787277 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.787246    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"openshift-service-ca.crt\""
Nov 03 17:49:06.787435 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.787318    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"kube-root-ca.crt\""
Nov 03 17:49:06.836674 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.836644    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"default-dockercfg-czx7h\""
Nov 03 17:49:06.836857 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.836703    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"simulator-w4-73a606-kserve-self-signed-certs\""
Nov 03 17:49:06.838904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.838865    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"]
Nov 03 17:49:06.859081 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.859055    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-dshm\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.859193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.859084    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-model-cache\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.859193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.859119    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tmp-dir\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.859266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.859243    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tls-certs\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.859337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.859324    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l26z\" (UniqueName: \"kubernetes.io/projected/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-kube-api-access-4l26z\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.859373 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.859353    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-home\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.960975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.960923    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-dshm\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.960984    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-model-cache\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.961030    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tmp-dir\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.961065    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tls-certs\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.961126    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l26z\" (UniqueName: \"kubernetes.io/projected/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-kube-api-access-4l26z\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.961161    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-home\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961500 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.961474    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-model-cache\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961675 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.961656    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tmp-dir\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.961924 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.961678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-home\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.963488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.963462    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-dshm\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.964214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:06.964195    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tls-certs\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:06.998159 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:49:06.998069    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:49:07.025974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:07.025934    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l26z\" (UniqueName: \"kubernetes.io/projected/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-kube-api-access-4l26z\") pod \"simulator-w4-73a606-kserve-86cdd6546-cz6hp\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") " pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:07.044315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:07.044277    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:07.424365 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:07.424338    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"]
Nov 03 17:49:07.426421 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:49:07.426392    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod419b56c2_9ba5_4f34_95bc_a2f45ae3d256.slice/crio-27a6bb11f3bf53f9b81a99c9b8cd09da543e8c77410bb7a4f7c506b06fd93699 WatchSource:0}: Error finding container 27a6bb11f3bf53f9b81a99c9b8cd09da543e8c77410bb7a4f7c506b06fd93699: Status 404 returned error can't find the container with id 27a6bb11f3bf53f9b81a99c9b8cd09da543e8c77410bb7a4f7c506b06fd93699
Nov 03 17:49:08.332173 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:08.332131    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" event={"ID":"419b56c2-9ba5-4f34-95bc-a2f45ae3d256","Type":"ContainerStarted","Data":"32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd"}
Nov 03 17:49:08.332173 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:08.332169    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" event={"ID":"419b56c2-9ba5-4f34-95bc-a2f45ae3d256","Type":"ContainerStarted","Data":"27a6bb11f3bf53f9b81a99c9b8cd09da543e8c77410bb7a4f7c506b06fd93699"}
Nov 03 17:49:08.453411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:08.453362    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podStartSLOduration=2.453348983 podStartE2EDuration="2.453348983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:08.453154074 +0000 UTC m=+927.912325059" watchObservedRunningTime="2026-11-03 17:49:08.453348983 +0000 UTC m=+927.912519967"
Nov 03 17:49:09.044843 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:09.044805    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:17.045300 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:17.045266    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:17.057606 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:17.057583    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:17.384122 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:17.384052    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:49:29.593469 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.593380    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"]
Nov 03 17:49:29.597151 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.597127    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.687112 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.687069    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-model-cache\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.687279 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.687137    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kkl\" (UniqueName: \"kubernetes.io/projected/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-kube-api-access-77kkl\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.687279 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.687251    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-home\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.687389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.687319    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tls-certs\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.687457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.687436    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tmp-dir\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.687535 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.687516    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-dshm\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.701780 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.701754    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"premium-w2-42c162-kserve-self-signed-certs\""
Nov 03 17:49:29.703554 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.703538    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"]
Nov 03 17:49:29.788635 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.788600    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-model-cache\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.788828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.788641    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kkl\" (UniqueName: \"kubernetes.io/projected/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-kube-api-access-77kkl\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.788828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.788699    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-home\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.788828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.788739    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tls-certs\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.788828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.788766    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tmp-dir\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.788828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.788800    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-dshm\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.789153 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.789129    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-model-cache\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.789222 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.789145    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-home\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.789282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.789218    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tmp-dir\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.791182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.791158    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-dshm\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.791355 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.791335    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tls-certs\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.852621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.852554    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kkl\" (UniqueName: \"kubernetes.io/projected/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-kube-api-access-77kkl\") pod \"premium-w2-42c162-kserve-6b8dcd7854-lh5qc\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") " pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:29.908815 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:29.908777    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:30.294732 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:30.289293    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"]
Nov 03 17:49:30.435224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:30.435189    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" event={"ID":"c66cfe66-7c3d-423c-b71e-b28fa4e47d75","Type":"ContainerStarted","Data":"df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048"}
Nov 03 17:49:30.435224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:30.435227    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" event={"ID":"c66cfe66-7c3d-423c-b71e-b28fa4e47d75","Type":"ContainerStarted","Data":"1ef783106e2b99f30829e1021e272b0b5c68711fceb1e2c219091142e1b4880e"}
Nov 03 17:49:30.557097 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:30.557001    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podStartSLOduration=1.556987621 podStartE2EDuration="1.556987621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:30.556568005 +0000 UTC m=+950.015738990" watchObservedRunningTime="2026-11-03 17:49:30.556987621 +0000 UTC m=+950.016158610"
Nov 03 17:49:30.909708 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:30.909638    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:36.992530 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:49:36.992484    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:49:39.909348 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:39.909309    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:39.928505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:39.928475    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:40.502489 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:40.502460    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:49:59.022765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.022732    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"]
Nov 03 17:49:59.026260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.026245    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.075591 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.075563    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.075706 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.075601    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.075749 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.075712    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rh5\" (UniqueName: \"kubernetes.io/projected/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-kube-api-access-c5rh5\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.081001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.080986    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:49:59.130837 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.130813    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:49:59.131000 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.130857    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:49:59.131000 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.130910    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:49:59.132812 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.132786    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"]
Nov 03 17:49:59.176461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.176435    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.176601 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.176490    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.176601 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.176585    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rh5\" (UniqueName: \"kubernetes.io/projected/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-kube-api-access-c5rh5\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.177044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.177023    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.179248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.179226    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.240666 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.240630    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rh5\" (UniqueName: \"kubernetes.io/projected/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-kube-api-access-c5rh5\") pod \"maas-api-revoke-sub-d394bd57-jthjz\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.339315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.339241    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:49:59.724228 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:49:59.724206    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"]
Nov 03 17:49:59.726333 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:49:59.726308    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce61bc3_ad99_4b69_a32a_afd8ffe2e728.slice/crio-4250d1ec5f087f9d5a7869b31e62222087dcd93d04cddb5aed9d032bc687490c WatchSource:0}: Error finding container 4250d1ec5f087f9d5a7869b31e62222087dcd93d04cddb5aed9d032bc687490c: Status 404 returned error can't find the container with id 4250d1ec5f087f9d5a7869b31e62222087dcd93d04cddb5aed9d032bc687490c
Nov 03 17:50:00.579195 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:00.579157    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz" event={"ID":"fce61bc3-ad99-4b69-a32a-afd8ffe2e728","Type":"ContainerStarted","Data":"4250d1ec5f087f9d5a7869b31e62222087dcd93d04cddb5aed9d032bc687490c"}
Nov 03 17:50:06.992599 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:50:06.992561    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:50:18.668527 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:18.668428    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz" event={"ID":"fce61bc3-ad99-4b69-a32a-afd8ffe2e728","Type":"ContainerStarted","Data":"f1ba214462c51b2e6ce2e80dc3a3f33bfbc673d4345aff71da52f12233097e90"}
Nov 03 17:50:18.789708 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:18.789648    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz" podStartSLOduration=2.448112482 podStartE2EDuration="20.789632786s" totalImagesPullingTime="18.341520304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:18.78800814 +0000 UTC m=+998.247179125" watchObservedRunningTime="2026-11-03 17:50:18.789632786 +0000 UTC m=+998.248803801"
Nov 03 17:50:19.675389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:19.675354    4493 generic.go:356] "Generic (PLEG): container finished" podID="fce61bc3-ad99-4b69-a32a-afd8ffe2e728" containerID="f1ba214462c51b2e6ce2e80dc3a3f33bfbc673d4345aff71da52f12233097e90" exitCode=0
Nov 03 17:50:19.675791 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:19.675412    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz" event={"ID":"fce61bc3-ad99-4b69-a32a-afd8ffe2e728","Type":"ContainerDied","Data":"f1ba214462c51b2e6ce2e80dc3a3f33bfbc673d4345aff71da52f12233097e90"}
Nov 03 17:50:20.816398 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.816375    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:50:20.898434 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.898403    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-xks-ca\") pod \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") "
Nov 03 17:50:20.898587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.898451    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-openshift-ca\") pod \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") "
Nov 03 17:50:20.898587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.898496    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-kube-api-access-c5rh5\" (UniqueName: \"kubernetes.io/projected/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-kube-api-access-c5rh5\") pod \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\" (UID: \"fce61bc3-ad99-4b69-a32a-afd8ffe2e728\") "
Nov 03 17:50:20.898914 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.898853    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-openshift-ca" pod "fce61bc3-ad99-4b69-a32a-afd8ffe2e728" (UID: "fce61bc3-ad99-4b69-a32a-afd8ffe2e728"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:50:20.900807 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.900783    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-xks-ca" pod "fce61bc3-ad99-4b69-a32a-afd8ffe2e728" (UID: "fce61bc3-ad99-4b69-a32a-afd8ffe2e728"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:50:20.900908 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.900891    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-kube-api-access-c5rh5" pod "fce61bc3-ad99-4b69-a32a-afd8ffe2e728" (UID: "fce61bc3-ad99-4b69-a32a-afd8ffe2e728"). InnerVolumeSpecName "kube-api-access-c5rh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:50:20.999866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.999834    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:20.999866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.999861    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:20.999866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:20.999895    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rh5\" (UniqueName: \"kubernetes.io/projected/fce61bc3-ad99-4b69-a32a-afd8ffe2e728-kube-api-access-c5rh5\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:21.686151 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:21.686115    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz" event={"ID":"fce61bc3-ad99-4b69-a32a-afd8ffe2e728","Type":"ContainerDied","Data":"4250d1ec5f087f9d5a7869b31e62222087dcd93d04cddb5aed9d032bc687490c"}
Nov 03 17:50:21.686323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:21.686187    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4250d1ec5f087f9d5a7869b31e62222087dcd93d04cddb5aed9d032bc687490c"
Nov 03 17:50:21.686323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:21.686130    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"
Nov 03 17:50:33.650768 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.650734    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"]
Nov 03 17:50:33.651181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.651133    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fce61bc3-ad99-4b69-a32a-afd8ffe2e728" containerName="revoke-keys"
Nov 03 17:50:33.651181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.651147    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce61bc3-ad99-4b69-a32a-afd8ffe2e728" containerName="revoke-keys"
Nov 03 17:50:33.651252 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.651229    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce61bc3-ad99-4b69-a32a-afd8ffe2e728" containerName="revoke-keys"
Nov 03 17:50:33.654247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.654231    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.728064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.728034    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-home\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.728226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.728073    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-dshm\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.728226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.728113    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34f5790-bf7f-432c-98dc-d2c206f1f307-tls-certs\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.728338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.728224    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4fb\" (UniqueName: \"kubernetes.io/projected/c34f5790-bf7f-432c-98dc-d2c206f1f307-kube-api-access-zh4fb\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.728412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.728392    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-model-cache\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.728475 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.728444    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-tmp-dir\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.758685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.758652    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"premium-w4-73a606-kserve-self-signed-certs\""
Nov 03 17:50:33.760561 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.760541    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"]
Nov 03 17:50:33.829939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.829897    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34f5790-bf7f-432c-98dc-d2c206f1f307-tls-certs\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.829939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.829944    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4fb\" (UniqueName: \"kubernetes.io/projected/c34f5790-bf7f-432c-98dc-d2c206f1f307-kube-api-access-zh4fb\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.830165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.830014    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-model-cache\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.830165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.830152    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-tmp-dir\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.830278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.830201    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-home\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.830278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.830251    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-dshm\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.830404 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.830384    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-model-cache\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.830526 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.830498    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-tmp-dir\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.830658 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.830506    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-home\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.832389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.832368    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-dshm\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.832494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.832479    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34f5790-bf7f-432c-98dc-d2c206f1f307-tls-certs\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.895278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.895238    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4fb\" (UniqueName: \"kubernetes.io/projected/c34f5790-bf7f-432c-98dc-d2c206f1f307-kube-api-access-zh4fb\") pod \"premium-w4-73a606-kserve-75b8ff487-qj8jk\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") " pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:33.965314 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:33.965270    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:34.347452 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:34.347431    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"]
Nov 03 17:50:34.350134 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:50:34.350108    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc34f5790_bf7f_432c_98dc_d2c206f1f307.slice/crio-b9b02165a834cc184f7233a9ccf3989529b3c6ce04d9847164b614fbd4f315b8 WatchSource:0}: Error finding container b9b02165a834cc184f7233a9ccf3989529b3c6ce04d9847164b614fbd4f315b8: Status 404 returned error can't find the container with id b9b02165a834cc184f7233a9ccf3989529b3c6ce04d9847164b614fbd4f315b8
Nov 03 17:50:34.746595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:34.746563    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" event={"ID":"c34f5790-bf7f-432c-98dc-d2c206f1f307","Type":"ContainerStarted","Data":"21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be"}
Nov 03 17:50:34.746595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:34.746601    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" event={"ID":"c34f5790-bf7f-432c-98dc-d2c206f1f307","Type":"ContainerStarted","Data":"b9b02165a834cc184f7233a9ccf3989529b3c6ce04d9847164b614fbd4f315b8"}
Nov 03 17:50:34.867564 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:34.867518    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podStartSLOduration=1.8674947240000002 podStartE2EDuration="1.867494724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:34.867116779 +0000 UTC m=+1014.326287764" watchObservedRunningTime="2026-11-03 17:50:34.867494724 +0000 UTC m=+1014.326665708"
Nov 03 17:50:34.965662 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:34.965629    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:35.867250 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.867212    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"]
Nov 03 17:50:35.870993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.870974    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:35.974591 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.974549    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:35.974768 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.974727    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:35.974831 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.974775    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88n8\" (UniqueName: \"kubernetes.io/projected/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-kube-api-access-d88n8\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:35.976054 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.976035    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:50:35.976165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.976125    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:50:35.976165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.976138    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:50:35.976262 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.976179    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:50:35.978602 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:35.978578    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"]
Nov 03 17:50:36.075890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.075850    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88n8\" (UniqueName: \"kubernetes.io/projected/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-kube-api-access-d88n8\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:36.076047 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.075938    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:36.076047 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.076033    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:36.076535 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.076509    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:36.078449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.078425    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:36.139623 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.139555    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88n8\" (UniqueName: \"kubernetes.io/projected/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-kube-api-access-d88n8\") pod \"maas-api-revoke-sub-0d1f8ba9-2knq7\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:36.190430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.190401    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:36.566083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.566060    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"]
Nov 03 17:50:36.567722 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:50:36.567692    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58bb9a7_c060_4c3e_8ff7_99d4c9884e62.slice/crio-aba65b7b7e69dfcefa8d6c7e36d19bd0c4d16a6c6b05418564bec8aa0cdfedc9 WatchSource:0}: Error finding container aba65b7b7e69dfcefa8d6c7e36d19bd0c4d16a6c6b05418564bec8aa0cdfedc9: Status 404 returned error can't find the container with id aba65b7b7e69dfcefa8d6c7e36d19bd0c4d16a6c6b05418564bec8aa0cdfedc9
Nov 03 17:50:36.758177 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:36.758146    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7" event={"ID":"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62","Type":"ContainerStarted","Data":"aba65b7b7e69dfcefa8d6c7e36d19bd0c4d16a6c6b05418564bec8aa0cdfedc9"}
Nov 03 17:50:36.991471 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:50:36.991438    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:50:37.764391 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:37.764363    4493 generic.go:356] "Generic (PLEG): container finished" podID="f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" containerID="3e46468cd59d9f410974b1a3d1490a1ae50dd61aaa2af3bd63851ed8921f540d" exitCode=0
Nov 03 17:50:37.764558 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:37.764409    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7" event={"ID":"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62","Type":"ContainerDied","Data":"3e46468cd59d9f410974b1a3d1490a1ae50dd61aaa2af3bd63851ed8921f540d"}
Nov 03 17:50:38.905695 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:38.905669    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:39.006621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.006594    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-xks-ca\") pod \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") "
Nov 03 17:50:39.006792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.006686    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-openshift-ca\") pod \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") "
Nov 03 17:50:39.006792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.006731    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-kube-api-access-d88n8\" (UniqueName: \"kubernetes.io/projected/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-kube-api-access-d88n8\") pod \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\" (UID: \"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62\") "
Nov 03 17:50:39.007057 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.007037    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-openshift-ca" pod "f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" (UID: "f58bb9a7-c060-4c3e-8ff7-99d4c9884e62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:50:39.007164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.007150    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:39.008865 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.008841    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-xks-ca" pod "f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" (UID: "f58bb9a7-c060-4c3e-8ff7-99d4c9884e62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:50:39.008988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.008857    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-kube-api-access-d88n8" pod "f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" (UID: "f58bb9a7-c060-4c3e-8ff7-99d4c9884e62"). InnerVolumeSpecName "kube-api-access-d88n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:50:39.107897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.107800    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:39.107897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.107825    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88n8\" (UniqueName: \"kubernetes.io/projected/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62-kube-api-access-d88n8\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:39.775703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.775669    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7" event={"ID":"f58bb9a7-c060-4c3e-8ff7-99d4c9884e62","Type":"ContainerDied","Data":"aba65b7b7e69dfcefa8d6c7e36d19bd0c4d16a6c6b05418564bec8aa0cdfedc9"}
Nov 03 17:50:39.775703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.775689    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"
Nov 03 17:50:39.775922 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:39.775721    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aba65b7b7e69dfcefa8d6c7e36d19bd0c4d16a6c6b05418564bec8aa0cdfedc9"
Nov 03 17:50:43.966083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:43.966037    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:43.978794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:43.978769    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:44.809319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:44.809293    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:50:45.132748 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.132676    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"]
Nov 03 17:50:45.133109 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.133055    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" containerName="revoke-keys"
Nov 03 17:50:45.133109 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.133065    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" containerName="revoke-keys"
Nov 03 17:50:45.133178 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.133149    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" containerName="revoke-keys"
Nov 03 17:50:45.142297 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.142267    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.197279 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.197244    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"]
Nov 03 17:50:45.197520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.197285    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:50:45.197520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.197324    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:50:45.197520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.197480    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:50:45.197780 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.197675    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:50:45.267354 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.267326    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.267526 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.267366    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.267526 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.267395    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpd7f\" (UniqueName: \"kubernetes.io/projected/7f19b936-6331-49d1-9eeb-38b3f85e4b63-kube-api-access-vpd7f\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.368184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.368151    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.368341 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.368194    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpd7f\" (UniqueName: \"kubernetes.io/projected/7f19b936-6331-49d1-9eeb-38b3f85e4b63-kube-api-access-vpd7f\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.368341 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.368268    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.368999 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.368977    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.370729 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.370714    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.432228 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.432167    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpd7f\" (UniqueName: \"kubernetes.io/projected/7f19b936-6331-49d1-9eeb-38b3f85e4b63-kube-api-access-vpd7f\") pod \"maas-api-revoke-sub-bc171688-7mcvj\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.457998 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.457971    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:45.836765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.836730    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"]
Nov 03 17:50:45.838955 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:50:45.838923    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f19b936_6331_49d1_9eeb_38b3f85e4b63.slice/crio-fd510608f665f4209b96d8e8f8cf88885633b39314b148f34df908a6a25e059c WatchSource:0}: Error finding container fd510608f665f4209b96d8e8f8cf88885633b39314b148f34df908a6a25e059c: Status 404 returned error can't find the container with id fd510608f665f4209b96d8e8f8cf88885633b39314b148f34df908a6a25e059c
Nov 03 17:50:45.839753 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.839724    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"]
Nov 03 17:50:45.845604 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.845583    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:45.903304 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.903276    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"kube-root-ca.crt\""
Nov 03 17:50:45.903434 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.903321    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"openshift-service-ca.crt\""
Nov 03 17:50:45.903653 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.903640    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"default-dockercfg-cqnvp\""
Nov 03 17:50:45.943312 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.943280    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"premium-w5-548ca1-kserve-self-signed-certs\""
Nov 03 17:50:45.950965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.950935    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"]
Nov 03 17:50:45.974766 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.974735    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-model-cache\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:45.974993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.974777    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-home\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:45.975058 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.975026    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tls-certs\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:45.975147 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.975122    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvbv\" (UniqueName: \"kubernetes.io/projected/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-kube-api-access-nfvbv\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:45.975188 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.975171    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-dshm\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:45.975228 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:45.975215    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tmp-dir\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076098 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076061    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tls-certs\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076098 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076110    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvbv\" (UniqueName: \"kubernetes.io/projected/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-kube-api-access-nfvbv\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076138    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-dshm\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076174    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tmp-dir\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076214    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-model-cache\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076586 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076475    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-home\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076653 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076632    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-model-cache\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076800 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076709    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tmp-dir\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.076800 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.076811    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-home\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.078954 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.078932    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tls-certs\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.079068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.078972    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-dshm\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.140383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.140347    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvbv\" (UniqueName: \"kubernetes.io/projected/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-kube-api-access-nfvbv\") pod \"premium-w5-548ca1-kserve-b7666bf99-j79kq\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") " pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.159414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.159386    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:46.556414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.556391    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"]
Nov 03 17:50:46.558532 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:50:46.558502    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod809fc6c4_b67d_4d4e_9225_39b4a2a6757d.slice/crio-d2819368c39aa837266bd850d12ee23a59e52408aa8ff439936ff45e68beb703 WatchSource:0}: Error finding container d2819368c39aa837266bd850d12ee23a59e52408aa8ff439936ff45e68beb703: Status 404 returned error can't find the container with id d2819368c39aa837266bd850d12ee23a59e52408aa8ff439936ff45e68beb703
Nov 03 17:50:46.811616 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.811533    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" event={"ID":"809fc6c4-b67d-4d4e-9225-39b4a2a6757d","Type":"ContainerStarted","Data":"14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233"}
Nov 03 17:50:46.811616 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.811583    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" event={"ID":"809fc6c4-b67d-4d4e-9225-39b4a2a6757d","Type":"ContainerStarted","Data":"d2819368c39aa837266bd850d12ee23a59e52408aa8ff439936ff45e68beb703"}
Nov 03 17:50:46.812963 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.812939    4493 generic.go:356] "Generic (PLEG): container finished" podID="7f19b936-6331-49d1-9eeb-38b3f85e4b63" containerID="3e5febf232357c7f4849040426cdcb67b52b0e87c7f11a204713c64790faeac9" exitCode=0
Nov 03 17:50:46.813076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.813015    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj" event={"ID":"7f19b936-6331-49d1-9eeb-38b3f85e4b63","Type":"ContainerDied","Data":"3e5febf232357c7f4849040426cdcb67b52b0e87c7f11a204713c64790faeac9"}
Nov 03 17:50:46.813076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.813056    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj" event={"ID":"7f19b936-6331-49d1-9eeb-38b3f85e4b63","Type":"ContainerStarted","Data":"fd510608f665f4209b96d8e8f8cf88885633b39314b148f34df908a6a25e059c"}
Nov 03 17:50:46.933020 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:46.932971    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podStartSLOduration=1.932956559 podStartE2EDuration="1.932956559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:46.932404946 +0000 UTC m=+1026.391575933" watchObservedRunningTime="2026-11-03 17:50:46.932956559 +0000 UTC m=+1026.392127540"
Nov 03 17:50:47.160319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:47.160240    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:47.960174 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:47.960151    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:48.097817 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.097732    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-openshift-ca\") pod \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") "
Nov 03 17:50:48.097817 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.097811    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f19b936-6331-49d1-9eeb-38b3f85e4b63-kube-api-access-vpd7f\" (UniqueName: \"kubernetes.io/projected/7f19b936-6331-49d1-9eeb-38b3f85e4b63-kube-api-access-vpd7f\") pod \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") "
Nov 03 17:50:48.098075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.097951    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-xks-ca\") pod \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\" (UID: \"7f19b936-6331-49d1-9eeb-38b3f85e4b63\") "
Nov 03 17:50:48.098247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.098220    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-openshift-ca" pod "7f19b936-6331-49d1-9eeb-38b3f85e4b63" (UID: "7f19b936-6331-49d1-9eeb-38b3f85e4b63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:50:48.098363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.098346    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:48.100251 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.100226    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-xks-ca" pod "7f19b936-6331-49d1-9eeb-38b3f85e4b63" (UID: "7f19b936-6331-49d1-9eeb-38b3f85e4b63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:50:48.100251 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.100234    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f19b936-6331-49d1-9eeb-38b3f85e4b63-kube-api-access-vpd7f" pod "7f19b936-6331-49d1-9eeb-38b3f85e4b63" (UID: "7f19b936-6331-49d1-9eeb-38b3f85e4b63"). InnerVolumeSpecName "kube-api-access-vpd7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:50:48.199144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.199112    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f19b936-6331-49d1-9eeb-38b3f85e4b63-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:48.199144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.199142    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpd7f\" (UniqueName: \"kubernetes.io/projected/7f19b936-6331-49d1-9eeb-38b3f85e4b63-kube-api-access-vpd7f\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:48.824214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.824182    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"
Nov 03 17:50:48.824390 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.824214    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj" event={"ID":"7f19b936-6331-49d1-9eeb-38b3f85e4b63","Type":"ContainerDied","Data":"fd510608f665f4209b96d8e8f8cf88885633b39314b148f34df908a6a25e059c"}
Nov 03 17:50:48.824390 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:48.824255    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd510608f665f4209b96d8e8f8cf88885633b39314b148f34df908a6a25e059c"
Nov 03 17:50:51.740588 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.740502    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"]
Nov 03 17:50:51.741174 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.741112    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7f19b936-6331-49d1-9eeb-38b3f85e4b63" containerName="revoke-keys"
Nov 03 17:50:51.741174 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.741131    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f19b936-6331-49d1-9eeb-38b3f85e4b63" containerName="revoke-keys"
Nov 03 17:50:51.741292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.741264    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f19b936-6331-49d1-9eeb-38b3f85e4b63" containerName="revoke-keys"
Nov 03 17:50:51.744603 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.744582    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.805193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.805161    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"]
Nov 03 17:50:51.836242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.836205    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thl8\" (UniqueName: \"kubernetes.io/projected/6210aa8f-2dc6-49f8-b499-81e61b9fea02-kube-api-access-7thl8\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.836432 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.836352    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.836432 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.836398    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.852119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.852091    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:50:51.852119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.852101    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:50:51.852307 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.852132    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:50:51.852307 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.852219    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:50:51.938055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.938011    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thl8\" (UniqueName: \"kubernetes.io/projected/6210aa8f-2dc6-49f8-b499-81e61b9fea02-kube-api-access-7thl8\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.938235 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.938103    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.938235 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.938134    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.938774 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.938743    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:51.940606 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:51.940585    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:52.004492 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:52.004416    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thl8\" (UniqueName: \"kubernetes.io/projected/6210aa8f-2dc6-49f8-b499-81e61b9fea02-kube-api-access-7thl8\") pod \"maas-api-revoke-sub-e6090a2b-rv4qz\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:52.057036 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:52.057010    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:52.435365 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:52.435337    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"]
Nov 03 17:50:52.437269 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:50:52.437239    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6210aa8f_2dc6_49f8_b499_81e61b9fea02.slice/crio-30f2041112536574ff9257df3f56d17a516c4c69b73d98ef804cb636f2112e48 WatchSource:0}: Error finding container 30f2041112536574ff9257df3f56d17a516c4c69b73d98ef804cb636f2112e48: Status 404 returned error can't find the container with id 30f2041112536574ff9257df3f56d17a516c4c69b73d98ef804cb636f2112e48
Nov 03 17:50:52.847565 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:52.847521    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz" event={"ID":"6210aa8f-2dc6-49f8-b499-81e61b9fea02","Type":"ContainerStarted","Data":"089e1422e0aea13b026d9d0ff844bf3ffa6b7c12f893ab65b8809ed48e414fa5"}
Nov 03 17:50:52.847963 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:52.847590    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz" event={"ID":"6210aa8f-2dc6-49f8-b499-81e61b9fea02","Type":"ContainerStarted","Data":"30f2041112536574ff9257df3f56d17a516c4c69b73d98ef804cb636f2112e48"}
Nov 03 17:50:52.968709 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:52.968663    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz" podStartSLOduration=1.96864951 podStartE2EDuration="1.96864951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:52.96713993 +0000 UTC m=+1032.426310918" watchObservedRunningTime="2026-11-03 17:50:52.96864951 +0000 UTC m=+1032.427820496"
Nov 03 17:50:53.213147 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:50:53.213117    4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6210aa8f_2dc6_49f8_b499_81e61b9fea02.slice/crio-089e1422e0aea13b026d9d0ff844bf3ffa6b7c12f893ab65b8809ed48e414fa5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 17:50:53.854248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:53.854218    4493 generic.go:356] "Generic (PLEG): container finished" podID="6210aa8f-2dc6-49f8-b499-81e61b9fea02" containerID="089e1422e0aea13b026d9d0ff844bf3ffa6b7c12f893ab65b8809ed48e414fa5" exitCode=0
Nov 03 17:50:53.854248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:53.854254    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz" event={"ID":"6210aa8f-2dc6-49f8-b499-81e61b9fea02","Type":"ContainerDied","Data":"089e1422e0aea13b026d9d0ff844bf3ffa6b7c12f893ab65b8809ed48e414fa5"}
Nov 03 17:50:55.003811 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.003785    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:55.169343 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.169271    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6210aa8f-2dc6-49f8-b499-81e61b9fea02-kube-api-access-7thl8\" (UniqueName: \"kubernetes.io/projected/6210aa8f-2dc6-49f8-b499-81e61b9fea02-kube-api-access-7thl8\") pod \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") "
Nov 03 17:50:55.169487 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.169408    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-xks-ca\") pod \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") "
Nov 03 17:50:55.169487 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.169450    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-openshift-ca\") pod \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\" (UID: \"6210aa8f-2dc6-49f8-b499-81e61b9fea02\") "
Nov 03 17:50:55.169840 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.169816    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-openshift-ca" pod "6210aa8f-2dc6-49f8-b499-81e61b9fea02" (UID: "6210aa8f-2dc6-49f8-b499-81e61b9fea02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:50:55.171577 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.171553    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6210aa8f-2dc6-49f8-b499-81e61b9fea02-kube-api-access-7thl8" pod "6210aa8f-2dc6-49f8-b499-81e61b9fea02" (UID: "6210aa8f-2dc6-49f8-b499-81e61b9fea02"). InnerVolumeSpecName "kube-api-access-7thl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:50:55.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.171571    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-xks-ca" pod "6210aa8f-2dc6-49f8-b499-81e61b9fea02" (UID: "6210aa8f-2dc6-49f8-b499-81e61b9fea02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:50:55.271178 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.271150    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:55.271178 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.271180    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thl8\" (UniqueName: \"kubernetes.io/projected/6210aa8f-2dc6-49f8-b499-81e61b9fea02-kube-api-access-7thl8\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:55.271347 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.271194    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6210aa8f-2dc6-49f8-b499-81e61b9fea02-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:55.864201 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.864169    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"
Nov 03 17:50:55.864380 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.864204    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz" event={"ID":"6210aa8f-2dc6-49f8-b499-81e61b9fea02","Type":"ContainerDied","Data":"30f2041112536574ff9257df3f56d17a516c4c69b73d98ef804cb636f2112e48"}
Nov 03 17:50:55.864380 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.864252    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f2041112536574ff9257df3f56d17a516c4c69b73d98ef804cb636f2112e48"
Nov 03 17:50:55.980128 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.980095    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"]
Nov 03 17:50:55.980646 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.980632    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6210aa8f-2dc6-49f8-b499-81e61b9fea02" containerName="revoke-keys"
Nov 03 17:50:55.980686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.980649    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6210aa8f-2dc6-49f8-b499-81e61b9fea02" containerName="revoke-keys"
Nov 03 17:50:55.980766 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.980756    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6210aa8f-2dc6-49f8-b499-81e61b9fea02" containerName="revoke-keys"
Nov 03 17:50:55.984125 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:55.984104    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.016888 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.016846    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"]
Nov 03 17:50:56.041689 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.041653    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:50:56.064379 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.064346    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:50:56.064379 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.064353    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:50:56.064548 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.064478    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:50:56.160457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.160379    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:56.173132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.173108    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:56.178518 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.178497    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.178586 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.178555    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmrx\" (UniqueName: \"kubernetes.io/projected/7fa2e595-aa5f-4e31-9298-9a269ba3b141-kube-api-access-fnmrx\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.178702 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.178681    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.279923 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.279865    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.280182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.279960    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmrx\" (UniqueName: \"kubernetes.io/projected/7fa2e595-aa5f-4e31-9298-9a269ba3b141-kube-api-access-fnmrx\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.280182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.280000    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.280539 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.280517    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.282442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.282421    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.344423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.344369    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmrx\" (UniqueName: \"kubernetes.io/projected/7fa2e595-aa5f-4e31-9298-9a269ba3b141-kube-api-access-fnmrx\") pod \"maas-api-revoke-sub-3447050f-662tr\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.596803 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.596765    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:56.880023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.879951    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:50:56.974367 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:56.974345    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"]
Nov 03 17:50:56.976187 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:50:56.976159    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa2e595_aa5f_4e31_9298_9a269ba3b141.slice/crio-41040998f41514aa2b80d15df360f68f485b9d9f420d23ac0b39010e0f868d14 WatchSource:0}: Error finding container 41040998f41514aa2b80d15df360f68f485b9d9f420d23ac0b39010e0f868d14: Status 404 returned error can't find the container with id 41040998f41514aa2b80d15df360f68f485b9d9f420d23ac0b39010e0f868d14
Nov 03 17:50:57.876264 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:57.876229    4493 generic.go:356] "Generic (PLEG): container finished" podID="7fa2e595-aa5f-4e31-9298-9a269ba3b141" containerID="064c6bf230ac563b7ea50972e4ae67e6e8a048c88fbf8a4616f19195cc31e3e1" exitCode=0
Nov 03 17:50:57.876686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:57.876285    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr" event={"ID":"7fa2e595-aa5f-4e31-9298-9a269ba3b141","Type":"ContainerDied","Data":"064c6bf230ac563b7ea50972e4ae67e6e8a048c88fbf8a4616f19195cc31e3e1"}
Nov 03 17:50:57.876686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:57.876367    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr" event={"ID":"7fa2e595-aa5f-4e31-9298-9a269ba3b141","Type":"ContainerStarted","Data":"41040998f41514aa2b80d15df360f68f485b9d9f420d23ac0b39010e0f868d14"}
Nov 03 17:50:59.021719 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.021695    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:59.108904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.108847    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-openshift-ca\") pod \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") "
Nov 03 17:50:59.109094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.108973    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa2e595-aa5f-4e31-9298-9a269ba3b141-kube-api-access-fnmrx\" (UniqueName: \"kubernetes.io/projected/7fa2e595-aa5f-4e31-9298-9a269ba3b141-kube-api-access-fnmrx\") pod \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") "
Nov 03 17:50:59.109094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.109014    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-xks-ca\") pod \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\" (UID: \"7fa2e595-aa5f-4e31-9298-9a269ba3b141\") "
Nov 03 17:50:59.109226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.109204    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-openshift-ca" pod "7fa2e595-aa5f-4e31-9298-9a269ba3b141" (UID: "7fa2e595-aa5f-4e31-9298-9a269ba3b141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:50:59.109383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.109364    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:59.111183 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.111160    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-xks-ca" pod "7fa2e595-aa5f-4e31-9298-9a269ba3b141" (UID: "7fa2e595-aa5f-4e31-9298-9a269ba3b141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:50:59.111299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.111223    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa2e595-aa5f-4e31-9298-9a269ba3b141-kube-api-access-fnmrx" pod "7fa2e595-aa5f-4e31-9298-9a269ba3b141" (UID: "7fa2e595-aa5f-4e31-9298-9a269ba3b141"). InnerVolumeSpecName "kube-api-access-fnmrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:50:59.209990 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.209914    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa2e595-aa5f-4e31-9298-9a269ba3b141-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:59.209990 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.209941    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnmrx\" (UniqueName: \"kubernetes.io/projected/7fa2e595-aa5f-4e31-9298-9a269ba3b141-kube-api-access-fnmrx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:50:59.888122 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.888092    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"
Nov 03 17:50:59.888296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.888120    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr" event={"ID":"7fa2e595-aa5f-4e31-9298-9a269ba3b141","Type":"ContainerDied","Data":"41040998f41514aa2b80d15df360f68f485b9d9f420d23ac0b39010e0f868d14"}
Nov 03 17:50:59.888296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:50:59.888167    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41040998f41514aa2b80d15df360f68f485b9d9f420d23ac0b39010e0f868d14"
Nov 03 17:51:06.993326 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:51:06.993275    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:51:08.088657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.088622    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"]
Nov 03 17:51:08.089071 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.089046    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7fa2e595-aa5f-4e31-9298-9a269ba3b141" containerName="revoke-keys"
Nov 03 17:51:08.089071 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.089057    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa2e595-aa5f-4e31-9298-9a269ba3b141" containerName="revoke-keys"
Nov 03 17:51:08.089151 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.089132    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa2e595-aa5f-4e31-9298-9a269ba3b141" containerName="revoke-keys"
Nov 03 17:51:08.092370 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.092352    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.150200 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.150161    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:51:08.150416 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.150228    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:51:08.152285 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.152251    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"]
Nov 03 17:51:08.182785 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.182744    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.182785 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.182788    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.182997 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.182923    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2lwb\" (UniqueName: \"kubernetes.io/projected/9f952732-7a64-4e0f-98d4-033d8c2c8fea-kube-api-access-c2lwb\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.200453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.200429    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:51:08.200856 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.200840    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:51:08.283585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.283555    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.283781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.283594    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.283781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.283746    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2lwb\" (UniqueName: \"kubernetes.io/projected/9f952732-7a64-4e0f-98d4-033d8c2c8fea-kube-api-access-c2lwb\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.284481 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.284460    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.286413 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.286394    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.347285 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.347221    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2lwb\" (UniqueName: \"kubernetes.io/projected/9f952732-7a64-4e0f-98d4-033d8c2c8fea-kube-api-access-c2lwb\") pod \"maas-api-revoke-sub-a97b68fc-zq87f\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.406005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.405976    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:08.791853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.791833    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"]
Nov 03 17:51:08.793663 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:08.793621    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f952732_7a64_4e0f_98d4_033d8c2c8fea.slice/crio-0a3b527077b279774097ecf4084e15a30ffd2ed807adfde5ced300f66a52d41e WatchSource:0}: Error finding container 0a3b527077b279774097ecf4084e15a30ffd2ed807adfde5ced300f66a52d41e: Status 404 returned error can't find the container with id 0a3b527077b279774097ecf4084e15a30ffd2ed807adfde5ced300f66a52d41e
Nov 03 17:51:08.934362 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:08.934331    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f" event={"ID":"9f952732-7a64-4e0f-98d4-033d8c2c8fea","Type":"ContainerStarted","Data":"0a3b527077b279774097ecf4084e15a30ffd2ed807adfde5ced300f66a52d41e"}
Nov 03 17:51:09.941091 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:09.941009    4493 generic.go:356] "Generic (PLEG): container finished" podID="9f952732-7a64-4e0f-98d4-033d8c2c8fea" containerID="66b0c996036b0f73520af48ac7640a51e3b4d87fedc2b7e403c966802fcb168b" exitCode=0
Nov 03 17:51:09.941445 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:09.941098    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f" event={"ID":"9f952732-7a64-4e0f-98d4-033d8c2c8fea","Type":"ContainerDied","Data":"66b0c996036b0f73520af48ac7640a51e3b4d87fedc2b7e403c966802fcb168b"}
Nov 03 17:51:11.085366 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.085334    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:11.110338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.110314    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-xks-ca\") pod \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") "
Nov 03 17:51:11.110486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.110345    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-openshift-ca\") pod \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") "
Nov 03 17:51:11.110486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.110457    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f952732-7a64-4e0f-98d4-033d8c2c8fea-kube-api-access-c2lwb\" (UniqueName: \"kubernetes.io/projected/9f952732-7a64-4e0f-98d4-033d8c2c8fea-kube-api-access-c2lwb\") pod \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\" (UID: \"9f952732-7a64-4e0f-98d4-033d8c2c8fea\") "
Nov 03 17:51:11.110718 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.110692    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-openshift-ca" pod "9f952732-7a64-4e0f-98d4-033d8c2c8fea" (UID: "9f952732-7a64-4e0f-98d4-033d8c2c8fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:11.112486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.112464    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-xks-ca" pod "9f952732-7a64-4e0f-98d4-033d8c2c8fea" (UID: "9f952732-7a64-4e0f-98d4-033d8c2c8fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:11.112600 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.112582    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f952732-7a64-4e0f-98d4-033d8c2c8fea-kube-api-access-c2lwb" pod "9f952732-7a64-4e0f-98d4-033d8c2c8fea" (UID: "9f952732-7a64-4e0f-98d4-033d8c2c8fea"). InnerVolumeSpecName "kube-api-access-c2lwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:11.212266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.212188    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2lwb\" (UniqueName: \"kubernetes.io/projected/9f952732-7a64-4e0f-98d4-033d8c2c8fea-kube-api-access-c2lwb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.212266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.212220    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.212266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.212231    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f952732-7a64-4e0f-98d4-033d8c2c8fea-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:11.953068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.953029    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f" event={"ID":"9f952732-7a64-4e0f-98d4-033d8c2c8fea","Type":"ContainerDied","Data":"0a3b527077b279774097ecf4084e15a30ffd2ed807adfde5ced300f66a52d41e"}
Nov 03 17:51:11.953254 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.953072    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"
Nov 03 17:51:11.953254 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:11.953094    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a3b527077b279774097ecf4084e15a30ffd2ed807adfde5ced300f66a52d41e"
Nov 03 17:51:18.795888 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.795846    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"]
Nov 03 17:51:18.796292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.796277    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9f952732-7a64-4e0f-98d4-033d8c2c8fea" containerName="revoke-keys"
Nov 03 17:51:18.796292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.796289    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f952732-7a64-4e0f-98d4-033d8c2c8fea" containerName="revoke-keys"
Nov 03 17:51:18.796395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.796353    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f952732-7a64-4e0f-98d4-033d8c2c8fea" containerName="revoke-keys"
Nov 03 17:51:18.799636 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.799611    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.855426 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.855394    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:51:18.884182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.884148    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.884364 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.884273    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.884364 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.884341    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6pw\" (UniqueName: \"kubernetes.io/projected/d2cdfe4c-f754-40bf-a994-30321ea9b764-kube-api-access-gn6pw\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.905529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.905484    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:51:18.905712 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.905549    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:51:18.905779 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.905710    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:51:18.908257 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.908236    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"]
Nov 03 17:51:18.985057 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.985021    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.985237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.985087    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.985237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.985130    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6pw\" (UniqueName: \"kubernetes.io/projected/d2cdfe4c-f754-40bf-a994-30321ea9b764-kube-api-access-gn6pw\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.985792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.985767    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.987775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.987727    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:18.991935 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.991912    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"]
Nov 03 17:51:18.995651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:18.995628    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.049244 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.049168    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6pw\" (UniqueName: \"kubernetes.io/projected/d2cdfe4c-f754-40bf-a994-30321ea9b764-kube-api-access-gn6pw\") pod \"maas-api-revoke-sub-6f82e893-l59lk\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:19.054924 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.054867    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"]
Nov 03 17:51:19.086329 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.086291    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.086506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.086346    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.086506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.086437    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6v5\" (UniqueName: \"kubernetes.io/projected/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-kube-api-access-6n6v5\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.112983 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.112946    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:19.187624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.187590    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.187769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.187631    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.187769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.187660    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6v5\" (UniqueName: \"kubernetes.io/projected/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-kube-api-access-6n6v5\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.188766 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.188682    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.190527 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.190499    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.251755 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.251717    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6v5\" (UniqueName: \"kubernetes.io/projected/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-kube-api-access-6n6v5\") pod \"maas-api-revoke-sub-d3dba8f0-htz6v\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.310012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.309931    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:19.497709 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.497683    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"]
Nov 03 17:51:19.499853 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:19.499826    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cdfe4c_f754_40bf_a994_30321ea9b764.slice/crio-96b86c1fb866089fe8bd9c023d0d8c004b059848c877f9db1d5df25276c0c169 WatchSource:0}: Error finding container 96b86c1fb866089fe8bd9c023d0d8c004b059848c877f9db1d5df25276c0c169: Status 404 returned error can't find the container with id 96b86c1fb866089fe8bd9c023d0d8c004b059848c877f9db1d5df25276c0c169
Nov 03 17:51:19.689894 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.689851    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"]
Nov 03 17:51:19.692048 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:19.692019    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b42601_ec89_4d26_8f21_d4cdb0e4623e.slice/crio-4386db68023a47ba28c2c9f4998eec3de846674f3fc529902a4b7111c016b7cd WatchSource:0}: Error finding container 4386db68023a47ba28c2c9f4998eec3de846674f3fc529902a4b7111c016b7cd: Status 404 returned error can't find the container with id 4386db68023a47ba28c2c9f4998eec3de846674f3fc529902a4b7111c016b7cd
Nov 03 17:51:19.996074 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.996035    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk" event={"ID":"d2cdfe4c-f754-40bf-a994-30321ea9b764","Type":"ContainerStarted","Data":"bda2b32d935d3286a56f412be64587133e135d737b9888dcd1c6dd97be4942c0"}
Nov 03 17:51:19.996517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.996103    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk" event={"ID":"d2cdfe4c-f754-40bf-a994-30321ea9b764","Type":"ContainerStarted","Data":"96b86c1fb866089fe8bd9c023d0d8c004b059848c877f9db1d5df25276c0c169"}
Nov 03 17:51:19.997474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:19.997447    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v" event={"ID":"d4b42601-ec89-4d26-8f21-d4cdb0e4623e","Type":"ContainerStarted","Data":"4386db68023a47ba28c2c9f4998eec3de846674f3fc529902a4b7111c016b7cd"}
Nov 03 17:51:20.116789 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:20.116728    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk" podStartSLOduration=2.116708255 podStartE2EDuration="2.116708255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:20.116113753 +0000 UTC m=+1059.575284757" watchObservedRunningTime="2026-11-03 17:51:20.116708255 +0000 UTC m=+1059.575879242"
Nov 03 17:51:21.004467 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:21.004434    4493 generic.go:356] "Generic (PLEG): container finished" podID="d2cdfe4c-f754-40bf-a994-30321ea9b764" containerID="bda2b32d935d3286a56f412be64587133e135d737b9888dcd1c6dd97be4942c0" exitCode=0
Nov 03 17:51:21.004903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:21.004520    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk" event={"ID":"d2cdfe4c-f754-40bf-a994-30321ea9b764","Type":"ContainerDied","Data":"bda2b32d935d3286a56f412be64587133e135d737b9888dcd1c6dd97be4942c0"}
Nov 03 17:51:21.006098 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:21.006075    4493 generic.go:356] "Generic (PLEG): container finished" podID="d4b42601-ec89-4d26-8f21-d4cdb0e4623e" containerID="4d5562f7cee78b02ca86ee11166a44b87167acd393b3485b5f3879c84aad192f" exitCode=0
Nov 03 17:51:21.006197 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:21.006153    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v" event={"ID":"d4b42601-ec89-4d26-8f21-d4cdb0e4623e","Type":"ContainerDied","Data":"4d5562f7cee78b02ca86ee11166a44b87167acd393b3485b5f3879c84aad192f"}
Nov 03 17:51:22.168989 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.168967    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:22.172055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.172038    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:22.216024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.215987    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cdfe4c-f754-40bf-a994-30321ea9b764-kube-api-access-gn6pw\" (UniqueName: \"kubernetes.io/projected/d2cdfe4c-f754-40bf-a994-30321ea9b764-kube-api-access-gn6pw\") pod \"d2cdfe4c-f754-40bf-a994-30321ea9b764\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") "
Nov 03 17:51:22.216024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.216026    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-xks-ca\") pod \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") "
Nov 03 17:51:22.216273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.216063    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-kube-api-access-6n6v5\" (UniqueName: \"kubernetes.io/projected/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-kube-api-access-6n6v5\") pod \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") "
Nov 03 17:51:22.216273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.216155    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-openshift-ca\") pod \"d2cdfe4c-f754-40bf-a994-30321ea9b764\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") "
Nov 03 17:51:22.216273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.216200    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-openshift-ca\") pod \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\" (UID: \"d4b42601-ec89-4d26-8f21-d4cdb0e4623e\") "
Nov 03 17:51:22.216273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.216259    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-xks-ca\") pod \"d2cdfe4c-f754-40bf-a994-30321ea9b764\" (UID: \"d2cdfe4c-f754-40bf-a994-30321ea9b764\") "
Nov 03 17:51:22.216549 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.216522    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-openshift-ca" pod "d2cdfe4c-f754-40bf-a994-30321ea9b764" (UID: "d2cdfe4c-f754-40bf-a994-30321ea9b764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:22.216614 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.216593    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-openshift-ca" pod "d4b42601-ec89-4d26-8f21-d4cdb0e4623e" (UID: "d4b42601-ec89-4d26-8f21-d4cdb0e4623e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:22.218554 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.218510    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-kube-api-access-6n6v5" pod "d4b42601-ec89-4d26-8f21-d4cdb0e4623e" (UID: "d4b42601-ec89-4d26-8f21-d4cdb0e4623e"). InnerVolumeSpecName "kube-api-access-6n6v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:22.218660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.218564    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-xks-ca" pod "d2cdfe4c-f754-40bf-a994-30321ea9b764" (UID: "d2cdfe4c-f754-40bf-a994-30321ea9b764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:22.218660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.218601    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-xks-ca" pod "d4b42601-ec89-4d26-8f21-d4cdb0e4623e" (UID: "d4b42601-ec89-4d26-8f21-d4cdb0e4623e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:22.218763 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.218747    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cdfe4c-f754-40bf-a994-30321ea9b764-kube-api-access-gn6pw" pod "d2cdfe4c-f754-40bf-a994-30321ea9b764" (UID: "d2cdfe4c-f754-40bf-a994-30321ea9b764"). InnerVolumeSpecName "kube-api-access-gn6pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:22.317529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.317444    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:22.317529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.317475    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:22.317529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.317491    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2cdfe4c-f754-40bf-a994-30321ea9b764-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:22.317529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.317503    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6pw\" (UniqueName: \"kubernetes.io/projected/d2cdfe4c-f754-40bf-a994-30321ea9b764-kube-api-access-gn6pw\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:22.317529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.317517    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:22.317529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:22.317531    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6v5\" (UniqueName: \"kubernetes.io/projected/d4b42601-ec89-4d26-8f21-d4cdb0e4623e-kube-api-access-6n6v5\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:23.017608 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:23.017522    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"
Nov 03 17:51:23.017608 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:23.017558    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v" event={"ID":"d4b42601-ec89-4d26-8f21-d4cdb0e4623e","Type":"ContainerDied","Data":"4386db68023a47ba28c2c9f4998eec3de846674f3fc529902a4b7111c016b7cd"}
Nov 03 17:51:23.017894 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:23.017640    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4386db68023a47ba28c2c9f4998eec3de846674f3fc529902a4b7111c016b7cd"
Nov 03 17:51:23.019024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:23.018993    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk" event={"ID":"d2cdfe4c-f754-40bf-a994-30321ea9b764","Type":"ContainerDied","Data":"96b86c1fb866089fe8bd9c023d0d8c004b059848c877f9db1d5df25276c0c169"}
Nov 03 17:51:23.019154 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:23.019032    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"
Nov 03 17:51:23.019154 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:23.019050    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96b86c1fb866089fe8bd9c023d0d8c004b059848c877f9db1d5df25276c0c169"
Nov 03 17:51:27.605263 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.605225    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"]
Nov 03 17:51:27.605687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.605673    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d2cdfe4c-f754-40bf-a994-30321ea9b764" containerName="revoke-keys"
Nov 03 17:51:27.605730 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.605688    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cdfe4c-f754-40bf-a994-30321ea9b764" containerName="revoke-keys"
Nov 03 17:51:27.605730 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.605702    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d4b42601-ec89-4d26-8f21-d4cdb0e4623e" containerName="revoke-keys"
Nov 03 17:51:27.605730 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.605707    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b42601-ec89-4d26-8f21-d4cdb0e4623e" containerName="revoke-keys"
Nov 03 17:51:27.605831 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.605763    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cdfe4c-f754-40bf-a994-30321ea9b764" containerName="revoke-keys"
Nov 03 17:51:27.605831 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.605772    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b42601-ec89-4d26-8f21-d4cdb0e4623e" containerName="revoke-keys"
Nov 03 17:51:27.608405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.608390    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.663551 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.663521    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:51:27.672384 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.672348    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16057974-779c-4803-b839-7a41487be851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.672595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.672573    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16057974-779c-4803-b839-7a41487be851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.672643 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.672627    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwxn\" (UniqueName: \"kubernetes.io/projected/16057974-779c-4803-b839-7a41487be851-kube-api-access-7jwxn\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.713464 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.713423    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:51:27.713464 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.713439    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:51:27.713704 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.713529    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:51:27.715930 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.715902    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"]
Nov 03 17:51:27.773409 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.773374    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwxn\" (UniqueName: \"kubernetes.io/projected/16057974-779c-4803-b839-7a41487be851-kube-api-access-7jwxn\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.773599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.773430    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16057974-779c-4803-b839-7a41487be851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.773599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.773553    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16057974-779c-4803-b839-7a41487be851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.774282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.774257    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16057974-779c-4803-b839-7a41487be851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.775957 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.775937    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16057974-779c-4803-b839-7a41487be851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.837643 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.837614    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwxn\" (UniqueName: \"kubernetes.io/projected/16057974-779c-4803-b839-7a41487be851-kube-api-access-7jwxn\") pod \"maas-api-revoke-sub-789eaf37-gxf6r\" (UID: \"16057974-779c-4803-b839-7a41487be851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:27.921035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:27.920958    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:28.296809 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:28.296784    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"]
Nov 03 17:51:28.299029 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:28.298993    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16057974_779c_4803_b839_7a41487be851.slice/crio-ca2b9f1fd98589b8db8e64753dbc6bd6f480a34d4c6d95bf1dd31e7985ad3aa9 WatchSource:0}: Error finding container ca2b9f1fd98589b8db8e64753dbc6bd6f480a34d4c6d95bf1dd31e7985ad3aa9: Status 404 returned error can't find the container with id ca2b9f1fd98589b8db8e64753dbc6bd6f480a34d4c6d95bf1dd31e7985ad3aa9
Nov 03 17:51:29.050933 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:29.050891    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r" event={"ID":"16057974-779c-4803-b839-7a41487be851","Type":"ContainerStarted","Data":"2ecddc094b8d7131115948a027fb5fb4ce6ff40d29021ad27292ba6034cd7ea8"}
Nov 03 17:51:29.051523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:29.050946    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r" event={"ID":"16057974-779c-4803-b839-7a41487be851","Type":"ContainerStarted","Data":"ca2b9f1fd98589b8db8e64753dbc6bd6f480a34d4c6d95bf1dd31e7985ad3aa9"}
Nov 03 17:51:29.173157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:29.173098    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r" podStartSLOduration=2.173081252 podStartE2EDuration="2.173081252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:29.171283997 +0000 UTC m=+1068.630454984" watchObservedRunningTime="2026-11-03 17:51:29.173081252 +0000 UTC m=+1068.632252238"
Nov 03 17:51:30.057144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:30.057108    4493 generic.go:356] "Generic (PLEG): container finished" podID="16057974-779c-4803-b839-7a41487be851" containerID="2ecddc094b8d7131115948a027fb5fb4ce6ff40d29021ad27292ba6034cd7ea8" exitCode=0
Nov 03 17:51:30.057569 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:30.057188    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r" event={"ID":"16057974-779c-4803-b839-7a41487be851","Type":"ContainerDied","Data":"2ecddc094b8d7131115948a027fb5fb4ce6ff40d29021ad27292ba6034cd7ea8"}
Nov 03 17:51:31.203799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.203771    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:31.311144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.311114    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16057974-779c-4803-b839-7a41487be851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16057974-779c-4803-b839-7a41487be851-maas-api-openshift-ca\") pod \"16057974-779c-4803-b839-7a41487be851\" (UID: \"16057974-779c-4803-b839-7a41487be851\") "
Nov 03 17:51:31.311321 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.311170    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16057974-779c-4803-b839-7a41487be851-kube-api-access-7jwxn\" (UniqueName: \"kubernetes.io/projected/16057974-779c-4803-b839-7a41487be851-kube-api-access-7jwxn\") pod \"16057974-779c-4803-b839-7a41487be851\" (UID: \"16057974-779c-4803-b839-7a41487be851\") "
Nov 03 17:51:31.311397 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.311375    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16057974-779c-4803-b839-7a41487be851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16057974-779c-4803-b839-7a41487be851-maas-api-xks-ca\") pod \"16057974-779c-4803-b839-7a41487be851\" (UID: \"16057974-779c-4803-b839-7a41487be851\") "
Nov 03 17:51:31.311532 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.311512    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16057974-779c-4803-b839-7a41487be851-maas-api-openshift-ca" pod "16057974-779c-4803-b839-7a41487be851" (UID: "16057974-779c-4803-b839-7a41487be851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:31.311703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.311688    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16057974-779c-4803-b839-7a41487be851-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:31.313584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.313527    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16057974-779c-4803-b839-7a41487be851-kube-api-access-7jwxn" pod "16057974-779c-4803-b839-7a41487be851" (UID: "16057974-779c-4803-b839-7a41487be851"). InnerVolumeSpecName "kube-api-access-7jwxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:31.313584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.313569    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16057974-779c-4803-b839-7a41487be851-maas-api-xks-ca" pod "16057974-779c-4803-b839-7a41487be851" (UID: "16057974-779c-4803-b839-7a41487be851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:31.412464 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.412424    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jwxn\" (UniqueName: \"kubernetes.io/projected/16057974-779c-4803-b839-7a41487be851-kube-api-access-7jwxn\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:31.412464 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:31.412457    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16057974-779c-4803-b839-7a41487be851-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:32.068184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:32.068157    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"
Nov 03 17:51:32.068364 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:32.068192    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r" event={"ID":"16057974-779c-4803-b839-7a41487be851","Type":"ContainerDied","Data":"ca2b9f1fd98589b8db8e64753dbc6bd6f480a34d4c6d95bf1dd31e7985ad3aa9"}
Nov 03 17:51:32.068364 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:32.068242    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2b9f1fd98589b8db8e64753dbc6bd6f480a34d4c6d95bf1dd31e7985ad3aa9"
Nov 03 17:51:35.858963 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.858923    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"]
Nov 03 17:51:35.859548 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.859527    4493 cpu_manager.go:414] "containerMap: removing container" podUID="16057974-779c-4803-b839-7a41487be851" containerName="revoke-keys"
Nov 03 17:51:35.859626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.859551    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16057974-779c-4803-b839-7a41487be851" containerName="revoke-keys"
Nov 03 17:51:35.859710 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.859697    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="16057974-779c-4803-b839-7a41487be851" containerName="revoke-keys"
Nov 03 17:51:35.863058 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.863034    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:35.924076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.924020    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"]
Nov 03 17:51:35.961246 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.961204    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:35.961414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.961342    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwn8f\" (UniqueName: \"kubernetes.io/projected/056435d9-5dee-4e5c-8508-ae9cfc456485-kube-api-access-cwn8f\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:35.961563 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.961542    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:35.970588 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.970558    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:51:35.970588 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.970556    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:51:35.970790 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.970644    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:51:35.970790 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:35.970708    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:51:36.062369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.062336    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:36.062585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.062421    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:36.062585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.062473    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwn8f\" (UniqueName: \"kubernetes.io/projected/056435d9-5dee-4e5c-8508-ae9cfc456485-kube-api-access-cwn8f\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:36.063101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.063080    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:36.064905 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.064884    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:36.127384 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.127308    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwn8f\" (UniqueName: \"kubernetes.io/projected/056435d9-5dee-4e5c-8508-ae9cfc456485-kube-api-access-cwn8f\") pod \"maas-api-revoke-sub-2c9bf09d-fqttg\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:36.177414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.177380    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:36.559333 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:36.559306    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"]
Nov 03 17:51:36.560851 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:36.560830    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056435d9_5dee_4e5c_8508_ae9cfc456485.slice/crio-1ac60443fa6793308e707c2cd251373da9bf6a1dc4af129f0a5a5f73e4440abe WatchSource:0}: Error finding container 1ac60443fa6793308e707c2cd251373da9bf6a1dc4af129f0a5a5f73e4440abe: Status 404 returned error can't find the container with id 1ac60443fa6793308e707c2cd251373da9bf6a1dc4af129f0a5a5f73e4440abe
Nov 03 17:51:36.991686 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:51:36.991647    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:51:37.098725 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.098640    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg" event={"ID":"056435d9-5dee-4e5c-8508-ae9cfc456485","Type":"ContainerStarted","Data":"fa24c3ded1b6bc193afabd74498ad517cf1799b4976c002182ad05e875f32ca2"}
Nov 03 17:51:37.098725 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.098700    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg" event={"ID":"056435d9-5dee-4e5c-8508-ae9cfc456485","Type":"ContainerStarted","Data":"1ac60443fa6793308e707c2cd251373da9bf6a1dc4af129f0a5a5f73e4440abe"}
Nov 03 17:51:37.174439 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.174410    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"]
Nov 03 17:51:37.178087 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.178072    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.219720 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.219672    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg" podStartSLOduration=2.219659708 podStartE2EDuration="2.219659708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:37.218749428 +0000 UTC m=+1076.677920417" watchObservedRunningTime="2026-11-03 17:51:37.219659708 +0000 UTC m=+1076.678830750"
Nov 03 17:51:37.237507 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.237477    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"]
Nov 03 17:51:37.273138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.273101    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.273289 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.273217    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.273289 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.273276    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbm46\" (UniqueName: \"kubernetes.io/projected/49ea6215-0ad8-4229-8c51-f6c1db7861d2-kube-api-access-rbm46\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.373898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.373779    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbm46\" (UniqueName: \"kubernetes.io/projected/49ea6215-0ad8-4229-8c51-f6c1db7861d2-kube-api-access-rbm46\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.373898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.373867    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.374107 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.373958    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.374747 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.374667    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.376385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.376360    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.437600 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.437572    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbm46\" (UniqueName: \"kubernetes.io/projected/49ea6215-0ad8-4229-8c51-f6c1db7861d2-kube-api-access-rbm46\") pod \"maas-api-revoke-sub-a150bb1b-4ztfc\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.490957 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.490924    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:37.869317 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:37.869290    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"]
Nov 03 17:51:37.871014 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:37.870983    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea6215_0ad8_4229_8c51_f6c1db7861d2.slice/crio-b4bcc0e9c4866bfbe63956245ff66f7395041d4b94a356cc3e97d9c46ce173bf WatchSource:0}: Error finding container b4bcc0e9c4866bfbe63956245ff66f7395041d4b94a356cc3e97d9c46ce173bf: Status 404 returned error can't find the container with id b4bcc0e9c4866bfbe63956245ff66f7395041d4b94a356cc3e97d9c46ce173bf
Nov 03 17:51:38.105451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:38.105407    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc" event={"ID":"49ea6215-0ad8-4229-8c51-f6c1db7861d2","Type":"ContainerStarted","Data":"b4bcc0e9c4866bfbe63956245ff66f7395041d4b94a356cc3e97d9c46ce173bf"}
Nov 03 17:51:38.106920 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:38.106887    4493 generic.go:356] "Generic (PLEG): container finished" podID="056435d9-5dee-4e5c-8508-ae9cfc456485" containerID="fa24c3ded1b6bc193afabd74498ad517cf1799b4976c002182ad05e875f32ca2" exitCode=0
Nov 03 17:51:38.107049 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:38.106921    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg" event={"ID":"056435d9-5dee-4e5c-8508-ae9cfc456485","Type":"ContainerDied","Data":"fa24c3ded1b6bc193afabd74498ad517cf1799b4976c002182ad05e875f32ca2"}
Nov 03 17:51:39.114278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.114185    4493 generic.go:356] "Generic (PLEG): container finished" podID="49ea6215-0ad8-4229-8c51-f6c1db7861d2" containerID="72088f6b72d5d27b38cf8538268d233f274510edd0eb30d306ac0b03bd003387" exitCode=0
Nov 03 17:51:39.114683 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.114276    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc" event={"ID":"49ea6215-0ad8-4229-8c51-f6c1db7861d2","Type":"ContainerDied","Data":"72088f6b72d5d27b38cf8538268d233f274510edd0eb30d306ac0b03bd003387"}
Nov 03 17:51:39.255206 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.255183    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:39.396029 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.395942    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-xks-ca\") pod \"056435d9-5dee-4e5c-8508-ae9cfc456485\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") "
Nov 03 17:51:39.396029 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.395995    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056435d9-5dee-4e5c-8508-ae9cfc456485-kube-api-access-cwn8f\" (UniqueName: \"kubernetes.io/projected/056435d9-5dee-4e5c-8508-ae9cfc456485-kube-api-access-cwn8f\") pod \"056435d9-5dee-4e5c-8508-ae9cfc456485\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") "
Nov 03 17:51:39.396254 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.396057    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-openshift-ca\") pod \"056435d9-5dee-4e5c-8508-ae9cfc456485\" (UID: \"056435d9-5dee-4e5c-8508-ae9cfc456485\") "
Nov 03 17:51:39.396439 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.396415    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-openshift-ca" pod "056435d9-5dee-4e5c-8508-ae9cfc456485" (UID: "056435d9-5dee-4e5c-8508-ae9cfc456485"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:39.398295 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.398261    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-xks-ca" pod "056435d9-5dee-4e5c-8508-ae9cfc456485" (UID: "056435d9-5dee-4e5c-8508-ae9cfc456485"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:39.398391 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.398330    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056435d9-5dee-4e5c-8508-ae9cfc456485-kube-api-access-cwn8f" pod "056435d9-5dee-4e5c-8508-ae9cfc456485" (UID: "056435d9-5dee-4e5c-8508-ae9cfc456485"). InnerVolumeSpecName "kube-api-access-cwn8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:39.497684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.497654    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:39.497684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.497682    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwn8f\" (UniqueName: \"kubernetes.io/projected/056435d9-5dee-4e5c-8508-ae9cfc456485-kube-api-access-cwn8f\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:39.497862 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:39.497698    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056435d9-5dee-4e5c-8508-ae9cfc456485-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:40.123314 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.123284    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"
Nov 03 17:51:40.123314 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.123290    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg" event={"ID":"056435d9-5dee-4e5c-8508-ae9cfc456485","Type":"ContainerDied","Data":"1ac60443fa6793308e707c2cd251373da9bf6a1dc4af129f0a5a5f73e4440abe"}
Nov 03 17:51:40.123842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.123352    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ac60443fa6793308e707c2cd251373da9bf6a1dc4af129f0a5a5f73e4440abe"
Nov 03 17:51:40.272233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.272205    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:40.411643 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.411555    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-xks-ca\") pod \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") "
Nov 03 17:51:40.411819 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.411650    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-openshift-ca\") pod \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") "
Nov 03 17:51:40.411901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.411864    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ea6215-0ad8-4229-8c51-f6c1db7861d2-kube-api-access-rbm46\" (UniqueName: \"kubernetes.io/projected/49ea6215-0ad8-4229-8c51-f6c1db7861d2-kube-api-access-rbm46\") pod \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\" (UID: \"49ea6215-0ad8-4229-8c51-f6c1db7861d2\") "
Nov 03 17:51:40.412075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.412047    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-openshift-ca" pod "49ea6215-0ad8-4229-8c51-f6c1db7861d2" (UID: "49ea6215-0ad8-4229-8c51-f6c1db7861d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:40.412194 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.412179    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:40.414412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.414374    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea6215-0ad8-4229-8c51-f6c1db7861d2-kube-api-access-rbm46" pod "49ea6215-0ad8-4229-8c51-f6c1db7861d2" (UID: "49ea6215-0ad8-4229-8c51-f6c1db7861d2"). InnerVolumeSpecName "kube-api-access-rbm46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:40.414545 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.414491    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-xks-ca" pod "49ea6215-0ad8-4229-8c51-f6c1db7861d2" (UID: "49ea6215-0ad8-4229-8c51-f6c1db7861d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:40.513179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.513143    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbm46\" (UniqueName: \"kubernetes.io/projected/49ea6215-0ad8-4229-8c51-f6c1db7861d2-kube-api-access-rbm46\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:40.513179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:40.513175    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ea6215-0ad8-4229-8c51-f6c1db7861d2-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:41.130863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:41.130779    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"
Nov 03 17:51:41.130863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:41.130809    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc" event={"ID":"49ea6215-0ad8-4229-8c51-f6c1db7861d2","Type":"ContainerDied","Data":"b4bcc0e9c4866bfbe63956245ff66f7395041d4b94a356cc3e97d9c46ce173bf"}
Nov 03 17:51:41.130863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:41.130859    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4bcc0e9c4866bfbe63956245ff66f7395041d4b94a356cc3e97d9c46ce173bf"
Nov 03 17:51:43.805557 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.805525    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"]
Nov 03 17:51:43.805948 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.805919    4493 cpu_manager.go:414] "containerMap: removing container" podUID="056435d9-5dee-4e5c-8508-ae9cfc456485" containerName="revoke-keys"
Nov 03 17:51:43.805948 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.805932    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056435d9-5dee-4e5c-8508-ae9cfc456485" containerName="revoke-keys"
Nov 03 17:51:43.805948 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.805945    4493 cpu_manager.go:414] "containerMap: removing container" podUID="49ea6215-0ad8-4229-8c51-f6c1db7861d2" containerName="revoke-keys"
Nov 03 17:51:43.805948 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.805950    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ea6215-0ad8-4229-8c51-f6c1db7861d2" containerName="revoke-keys"
Nov 03 17:51:43.806101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.806031    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ea6215-0ad8-4229-8c51-f6c1db7861d2" containerName="revoke-keys"
Nov 03 17:51:43.806101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.806044    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="056435d9-5dee-4e5c-8508-ae9cfc456485" containerName="revoke-keys"
Nov 03 17:51:43.809278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.809261    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:43.869122 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.869088    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"]
Nov 03 17:51:43.915937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.915907    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:51:43.915937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.915926    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:51:43.915937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.915907    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:51:43.916192 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.915976    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:51:43.945722 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.945687    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:43.945925 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.945727    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:43.945974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:43.945943    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsg7\" (UniqueName: \"kubernetes.io/projected/e841d75c-cff9-4ed7-9332-d40952b3813b-kube-api-access-6nsg7\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.046961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.046911    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.046961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.046966    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.047234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.047044    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsg7\" (UniqueName: \"kubernetes.io/projected/e841d75c-cff9-4ed7-9332-d40952b3813b-kube-api-access-6nsg7\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.047659 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.047637    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.049559 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.049531    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.111355 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.111280    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsg7\" (UniqueName: \"kubernetes.io/projected/e841d75c-cff9-4ed7-9332-d40952b3813b-kube-api-access-6nsg7\") pod \"maas-api-revoke-sub-629bf923-429zm\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.122555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.122523    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:44.505610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:44.505578    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"]
Nov 03 17:51:44.510820 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:44.510786    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode841d75c_cff9_4ed7_9332_d40952b3813b.slice/crio-57bdfa1139608c64513063c55a78ee038d4f16f1b99223116e6a4ede327d22fb WatchSource:0}: Error finding container 57bdfa1139608c64513063c55a78ee038d4f16f1b99223116e6a4ede327d22fb: Status 404 returned error can't find the container with id 57bdfa1139608c64513063c55a78ee038d4f16f1b99223116e6a4ede327d22fb
Nov 03 17:51:45.154424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:45.154300    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm" event={"ID":"e841d75c-cff9-4ed7-9332-d40952b3813b","Type":"ContainerStarted","Data":"3bb053f175f3dba2ec00247fee4118ac53ddae070c17ccfa3da7854eaac8dce3"}
Nov 03 17:51:45.154424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:45.154395    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm" event={"ID":"e841d75c-cff9-4ed7-9332-d40952b3813b","Type":"ContainerStarted","Data":"57bdfa1139608c64513063c55a78ee038d4f16f1b99223116e6a4ede327d22fb"}
Nov 03 17:51:45.275603 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:45.275545    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm" podStartSLOduration=2.275529064 podStartE2EDuration="2.275529064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:45.274199966 +0000 UTC m=+1084.733370951" watchObservedRunningTime="2026-11-03 17:51:45.275529064 +0000 UTC m=+1084.734700049"
Nov 03 17:51:46.162010 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.161974    4493 generic.go:356] "Generic (PLEG): container finished" podID="e841d75c-cff9-4ed7-9332-d40952b3813b" containerID="3bb053f175f3dba2ec00247fee4118ac53ddae070c17ccfa3da7854eaac8dce3" exitCode=0
Nov 03 17:51:46.162421 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.162068    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm" event={"ID":"e841d75c-cff9-4ed7-9332-d40952b3813b","Type":"ContainerDied","Data":"3bb053f175f3dba2ec00247fee4118ac53ddae070c17ccfa3da7854eaac8dce3"}
Nov 03 17:51:46.614002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.613961    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"]
Nov 03 17:51:46.617672 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.617648    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.675255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.675218    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.675442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.675266    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.675442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.675350    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q467m\" (UniqueName: \"kubernetes.io/projected/61e687f7-eac3-4317-949d-234a0bbdccb8-kube-api-access-q467m\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.677830 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.677803    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"]
Nov 03 17:51:46.776068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.776024    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q467m\" (UniqueName: \"kubernetes.io/projected/61e687f7-eac3-4317-949d-234a0bbdccb8-kube-api-access-q467m\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.776365 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.776310    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.776527 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.776398    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.777083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.777059    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.778959 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.778929    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.840184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.840149    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q467m\" (UniqueName: \"kubernetes.io/projected/61e687f7-eac3-4317-949d-234a0bbdccb8-kube-api-access-q467m\") pod \"maas-api-revoke-sub-3e3e283b-bldmm\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:46.929937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:46.929849    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:47.309281 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.309251    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"]
Nov 03 17:51:47.310911 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:47.310862    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e687f7_eac3_4317_949d_234a0bbdccb8.slice/crio-99d040338c6c5c8ea5c7c93d1c3b5911d529817c593309dca1c09fc5e6cc579b WatchSource:0}: Error finding container 99d040338c6c5c8ea5c7c93d1c3b5911d529817c593309dca1c09fc5e6cc579b: Status 404 returned error can't find the container with id 99d040338c6c5c8ea5c7c93d1c3b5911d529817c593309dca1c09fc5e6cc579b
Nov 03 17:51:47.312463 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.312438    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:47.383376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.383340    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e841d75c-cff9-4ed7-9332-d40952b3813b-kube-api-access-6nsg7\" (UniqueName: \"kubernetes.io/projected/e841d75c-cff9-4ed7-9332-d40952b3813b-kube-api-access-6nsg7\") pod \"e841d75c-cff9-4ed7-9332-d40952b3813b\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") "
Nov 03 17:51:47.383578 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.383501    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-xks-ca\") pod \"e841d75c-cff9-4ed7-9332-d40952b3813b\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") "
Nov 03 17:51:47.383578 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.383529    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-openshift-ca\") pod \"e841d75c-cff9-4ed7-9332-d40952b3813b\" (UID: \"e841d75c-cff9-4ed7-9332-d40952b3813b\") "
Nov 03 17:51:47.383999 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.383969    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-openshift-ca" pod "e841d75c-cff9-4ed7-9332-d40952b3813b" (UID: "e841d75c-cff9-4ed7-9332-d40952b3813b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:47.385675 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.385651    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-xks-ca" pod "e841d75c-cff9-4ed7-9332-d40952b3813b" (UID: "e841d75c-cff9-4ed7-9332-d40952b3813b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:47.385767 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.385688    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e841d75c-cff9-4ed7-9332-d40952b3813b-kube-api-access-6nsg7" pod "e841d75c-cff9-4ed7-9332-d40952b3813b" (UID: "e841d75c-cff9-4ed7-9332-d40952b3813b"). InnerVolumeSpecName "kube-api-access-6nsg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:47.484679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.484597    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:47.484679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.484628    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e841d75c-cff9-4ed7-9332-d40952b3813b-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:47.484679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:47.484641    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nsg7\" (UniqueName: \"kubernetes.io/projected/e841d75c-cff9-4ed7-9332-d40952b3813b-kube-api-access-6nsg7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:48.173562 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:48.173478    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm" event={"ID":"e841d75c-cff9-4ed7-9332-d40952b3813b","Type":"ContainerDied","Data":"57bdfa1139608c64513063c55a78ee038d4f16f1b99223116e6a4ede327d22fb"}
Nov 03 17:51:48.173700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:48.173500    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"
Nov 03 17:51:48.173700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:48.173579    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57bdfa1139608c64513063c55a78ee038d4f16f1b99223116e6a4ede327d22fb"
Nov 03 17:51:48.175038 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:48.175012    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm" event={"ID":"61e687f7-eac3-4317-949d-234a0bbdccb8","Type":"ContainerStarted","Data":"2fd76b63ef326bbc5f3d9573de7425852410d9072d7df1a206db2bf441bff693"}
Nov 03 17:51:48.175153 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:48.175062    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm" event={"ID":"61e687f7-eac3-4317-949d-234a0bbdccb8","Type":"ContainerStarted","Data":"99d040338c6c5c8ea5c7c93d1c3b5911d529817c593309dca1c09fc5e6cc579b"}
Nov 03 17:51:48.295800 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:48.295747    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm" podStartSLOduration=2.2957348 podStartE2EDuration="2.2957348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:48.294566933 +0000 UTC m=+1087.753737959" watchObservedRunningTime="2026-11-03 17:51:48.2957348 +0000 UTC m=+1087.754905794"
Nov 03 17:51:49.181798 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:49.181766    4493 generic.go:356] "Generic (PLEG): container finished" podID="61e687f7-eac3-4317-949d-234a0bbdccb8" containerID="2fd76b63ef326bbc5f3d9573de7425852410d9072d7df1a206db2bf441bff693" exitCode=0
Nov 03 17:51:49.182235 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:49.181855    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm" event={"ID":"61e687f7-eac3-4317-949d-234a0bbdccb8","Type":"ContainerDied","Data":"2fd76b63ef326bbc5f3d9573de7425852410d9072d7df1a206db2bf441bff693"}
Nov 03 17:51:50.327861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.327836    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:50.413520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.413489    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e687f7-eac3-4317-949d-234a0bbdccb8-kube-api-access-q467m\" (UniqueName: \"kubernetes.io/projected/61e687f7-eac3-4317-949d-234a0bbdccb8-kube-api-access-q467m\") pod \"61e687f7-eac3-4317-949d-234a0bbdccb8\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") "
Nov 03 17:51:50.413672 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.413591    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-xks-ca\") pod \"61e687f7-eac3-4317-949d-234a0bbdccb8\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") "
Nov 03 17:51:50.413672 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.413633    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-openshift-ca\") pod \"61e687f7-eac3-4317-949d-234a0bbdccb8\" (UID: \"61e687f7-eac3-4317-949d-234a0bbdccb8\") "
Nov 03 17:51:50.414010 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.413981    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-openshift-ca" pod "61e687f7-eac3-4317-949d-234a0bbdccb8" (UID: "61e687f7-eac3-4317-949d-234a0bbdccb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:50.415752 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.415731    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-xks-ca" pod "61e687f7-eac3-4317-949d-234a0bbdccb8" (UID: "61e687f7-eac3-4317-949d-234a0bbdccb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:50.415838 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.415781    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e687f7-eac3-4317-949d-234a0bbdccb8-kube-api-access-q467m" pod "61e687f7-eac3-4317-949d-234a0bbdccb8" (UID: "61e687f7-eac3-4317-949d-234a0bbdccb8"). InnerVolumeSpecName "kube-api-access-q467m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:50.514889 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.514846    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q467m\" (UniqueName: \"kubernetes.io/projected/61e687f7-eac3-4317-949d-234a0bbdccb8-kube-api-access-q467m\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:50.514889 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.514892    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:50.515094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:50.514904    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e687f7-eac3-4317-949d-234a0bbdccb8-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:51.195383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:51.195352    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm" event={"ID":"61e687f7-eac3-4317-949d-234a0bbdccb8","Type":"ContainerDied","Data":"99d040338c6c5c8ea5c7c93d1c3b5911d529817c593309dca1c09fc5e6cc579b"}
Nov 03 17:51:51.195383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:51.195375    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"
Nov 03 17:51:51.195631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:51.195406    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d040338c6c5c8ea5c7c93d1c3b5911d529817c593309dca1c09fc5e6cc579b"
Nov 03 17:51:53.971816 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.971781    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"]
Nov 03 17:51:53.972272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.972195    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e841d75c-cff9-4ed7-9332-d40952b3813b" containerName="revoke-keys"
Nov 03 17:51:53.972272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.972208    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e841d75c-cff9-4ed7-9332-d40952b3813b" containerName="revoke-keys"
Nov 03 17:51:53.972272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.972243    4493 cpu_manager.go:414] "containerMap: removing container" podUID="61e687f7-eac3-4317-949d-234a0bbdccb8" containerName="revoke-keys"
Nov 03 17:51:53.972272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.972248    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e687f7-eac3-4317-949d-234a0bbdccb8" containerName="revoke-keys"
Nov 03 17:51:53.972404 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.972303    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e841d75c-cff9-4ed7-9332-d40952b3813b" containerName="revoke-keys"
Nov 03 17:51:53.972404 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.972316    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e687f7-eac3-4317-949d-234a0bbdccb8" containerName="revoke-keys"
Nov 03 17:51:53.975940 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:53.975915    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.035628 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.035594    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"]
Nov 03 17:51:54.051286 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.051256    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.051425 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.051380    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.051466 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.051435    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s5s\" (UniqueName: \"kubernetes.io/projected/82ec2497-20d5-42bd-ad51-827fd0eb926c-kube-api-access-h5s5s\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.082902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.082842    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:51:54.082902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.082859    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:51:54.082902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.082899    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:51:54.083152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.082920    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:51:54.152282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.152247    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.152438 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.152319    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s5s\" (UniqueName: \"kubernetes.io/projected/82ec2497-20d5-42bd-ad51-827fd0eb926c-kube-api-access-h5s5s\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.152438 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.152398    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.153153 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.153079    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.154806 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.154785    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.216289 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.216259    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s5s\" (UniqueName: \"kubernetes.io/projected/82ec2497-20d5-42bd-ad51-827fd0eb926c-kube-api-access-h5s5s\") pod \"maas-api-revoke-sub-78815193-w7sh4\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.289960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.289860    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:54.670930 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.670903    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"]
Nov 03 17:51:54.673100 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:54.673071    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ec2497_20d5_42bd_ad51_827fd0eb926c.slice/crio-845f247f3a52ece24c2807ca6378e33fe290c7b2359f6183ca46c8e55d3bb242 WatchSource:0}: Error finding container 845f247f3a52ece24c2807ca6378e33fe290c7b2359f6183ca46c8e55d3bb242: Status 404 returned error can't find the container with id 845f247f3a52ece24c2807ca6378e33fe290c7b2359f6183ca46c8e55d3bb242
Nov 03 17:51:54.907402 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.907364    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"]
Nov 03 17:51:54.912260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.912237    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:54.961376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.961337    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsnfz\" (UniqueName: \"kubernetes.io/projected/56402ba4-5419-40d2-b92e-2cf682432625-kube-api-access-wsnfz\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:54.961575 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.961411    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:54.961647 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.961609    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:54.971625 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:54.971524    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"]
Nov 03 17:51:55.062525 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.062487    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:55.062950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.062567    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsnfz\" (UniqueName: \"kubernetes.io/projected/56402ba4-5419-40d2-b92e-2cf682432625-kube-api-access-wsnfz\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:55.062950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.062608    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:55.063277 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.063257    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:55.065107 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.065089    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:55.126697 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.126658    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsnfz\" (UniqueName: \"kubernetes.io/projected/56402ba4-5419-40d2-b92e-2cf682432625-kube-api-access-wsnfz\") pod \"maas-api-revoke-sub-3793979c-6548s\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:55.220994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.220961    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4" event={"ID":"82ec2497-20d5-42bd-ad51-827fd0eb926c","Type":"ContainerStarted","Data":"6f86a8979ce479824ed92ae3b047690e4352faea6d1c6a890a6b180d9e06b372"}
Nov 03 17:51:55.221192 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.221013    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4" event={"ID":"82ec2497-20d5-42bd-ad51-827fd0eb926c","Type":"ContainerStarted","Data":"845f247f3a52ece24c2807ca6378e33fe290c7b2359f6183ca46c8e55d3bb242"}
Nov 03 17:51:55.225377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.225356    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:55.343900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.343825    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4" podStartSLOduration=2.34381039 podStartE2EDuration="2.34381039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:55.341024079 +0000 UTC m=+1094.800195068" watchObservedRunningTime="2026-11-03 17:51:55.34381039 +0000 UTC m=+1094.802981376"
Nov 03 17:51:55.604800 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:55.604773    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"]
Nov 03 17:51:55.605837 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:51:55.605801    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56402ba4_5419_40d2_b92e_2cf682432625.slice/crio-8b4c6c976f4685b9ff61b64bca6a3aec6db61efb95f87054f1fdb0909c92a64c WatchSource:0}: Error finding container 8b4c6c976f4685b9ff61b64bca6a3aec6db61efb95f87054f1fdb0909c92a64c: Status 404 returned error can't find the container with id 8b4c6c976f4685b9ff61b64bca6a3aec6db61efb95f87054f1fdb0909c92a64c
Nov 03 17:51:56.228449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:56.228408    4493 generic.go:356] "Generic (PLEG): container finished" podID="82ec2497-20d5-42bd-ad51-827fd0eb926c" containerID="6f86a8979ce479824ed92ae3b047690e4352faea6d1c6a890a6b180d9e06b372" exitCode=0
Nov 03 17:51:56.228842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:56.228486    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4" event={"ID":"82ec2497-20d5-42bd-ad51-827fd0eb926c","Type":"ContainerDied","Data":"6f86a8979ce479824ed92ae3b047690e4352faea6d1c6a890a6b180d9e06b372"}
Nov 03 17:51:56.230165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:56.230134    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s" event={"ID":"56402ba4-5419-40d2-b92e-2cf682432625","Type":"ContainerStarted","Data":"532ca4b8a475f6676625637f4942f515cc316b07d462db78c1a3f5f6b8e4851e"}
Nov 03 17:51:56.230278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:56.230179    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s" event={"ID":"56402ba4-5419-40d2-b92e-2cf682432625","Type":"ContainerStarted","Data":"8b4c6c976f4685b9ff61b64bca6a3aec6db61efb95f87054f1fdb0909c92a64c"}
Nov 03 17:51:56.470294 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:56.470239    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s" podStartSLOduration=2.470225024 podStartE2EDuration="2.470225024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:56.467955443 +0000 UTC m=+1095.927126429" watchObservedRunningTime="2026-11-03 17:51:56.470225024 +0000 UTC m=+1095.929396009"
Nov 03 17:51:57.237337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.237306    4493 generic.go:356] "Generic (PLEG): container finished" podID="56402ba4-5419-40d2-b92e-2cf682432625" containerID="532ca4b8a475f6676625637f4942f515cc316b07d462db78c1a3f5f6b8e4851e" exitCode=0
Nov 03 17:51:57.237781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.237396    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s" event={"ID":"56402ba4-5419-40d2-b92e-2cf682432625","Type":"ContainerDied","Data":"532ca4b8a475f6676625637f4942f515cc316b07d462db78c1a3f5f6b8e4851e"}
Nov 03 17:51:57.379783 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.379757    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:57.488786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.488694    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ec2497-20d5-42bd-ad51-827fd0eb926c-kube-api-access-h5s5s\" (UniqueName: \"kubernetes.io/projected/82ec2497-20d5-42bd-ad51-827fd0eb926c-kube-api-access-h5s5s\") pod \"82ec2497-20d5-42bd-ad51-827fd0eb926c\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") "
Nov 03 17:51:57.488983 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.488801    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-xks-ca\") pod \"82ec2497-20d5-42bd-ad51-827fd0eb926c\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") "
Nov 03 17:51:57.488983 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.488905    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-openshift-ca\") pod \"82ec2497-20d5-42bd-ad51-827fd0eb926c\" (UID: \"82ec2497-20d5-42bd-ad51-827fd0eb926c\") "
Nov 03 17:51:57.489285 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.489255    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-openshift-ca" pod "82ec2497-20d5-42bd-ad51-827fd0eb926c" (UID: "82ec2497-20d5-42bd-ad51-827fd0eb926c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:57.491080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.491059    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ec2497-20d5-42bd-ad51-827fd0eb926c-kube-api-access-h5s5s" pod "82ec2497-20d5-42bd-ad51-827fd0eb926c" (UID: "82ec2497-20d5-42bd-ad51-827fd0eb926c"). InnerVolumeSpecName "kube-api-access-h5s5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:57.491187 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.491122    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-xks-ca" pod "82ec2497-20d5-42bd-ad51-827fd0eb926c" (UID: "82ec2497-20d5-42bd-ad51-827fd0eb926c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:57.590585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.590536    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:57.590585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.590578    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5s5s\" (UniqueName: \"kubernetes.io/projected/82ec2497-20d5-42bd-ad51-827fd0eb926c-kube-api-access-h5s5s\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:57.590585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:57.590592    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82ec2497-20d5-42bd-ad51-827fd0eb926c-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:58.244715 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.244683    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"
Nov 03 17:51:58.245139 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.244722    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4" event={"ID":"82ec2497-20d5-42bd-ad51-827fd0eb926c","Type":"ContainerDied","Data":"845f247f3a52ece24c2807ca6378e33fe290c7b2359f6183ca46c8e55d3bb242"}
Nov 03 17:51:58.245139 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.244770    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="845f247f3a52ece24c2807ca6378e33fe290c7b2359f6183ca46c8e55d3bb242"
Nov 03 17:51:58.384447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.384419    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:58.498687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.498602    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-xks-ca\") pod \"56402ba4-5419-40d2-b92e-2cf682432625\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") "
Nov 03 17:51:58.498864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.498698    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56402ba4-5419-40d2-b92e-2cf682432625-kube-api-access-wsnfz\" (UniqueName: \"kubernetes.io/projected/56402ba4-5419-40d2-b92e-2cf682432625-kube-api-access-wsnfz\") pod \"56402ba4-5419-40d2-b92e-2cf682432625\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") "
Nov 03 17:51:58.498864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.498741    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-openshift-ca\") pod \"56402ba4-5419-40d2-b92e-2cf682432625\" (UID: \"56402ba4-5419-40d2-b92e-2cf682432625\") "
Nov 03 17:51:58.499194 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.499168    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-openshift-ca" pod "56402ba4-5419-40d2-b92e-2cf682432625" (UID: "56402ba4-5419-40d2-b92e-2cf682432625"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:51:58.500993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.500966    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56402ba4-5419-40d2-b92e-2cf682432625-kube-api-access-wsnfz" pod "56402ba4-5419-40d2-b92e-2cf682432625" (UID: "56402ba4-5419-40d2-b92e-2cf682432625"). InnerVolumeSpecName "kube-api-access-wsnfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:51:58.501065 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.500972    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-xks-ca" pod "56402ba4-5419-40d2-b92e-2cf682432625" (UID: "56402ba4-5419-40d2-b92e-2cf682432625"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:51:58.600191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.600152    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:58.600191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.600185    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsnfz\" (UniqueName: \"kubernetes.io/projected/56402ba4-5419-40d2-b92e-2cf682432625-kube-api-access-wsnfz\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:58.600191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:58.600197    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56402ba4-5419-40d2-b92e-2cf682432625-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:51:59.252186 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:59.252156    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"
Nov 03 17:51:59.252616 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:59.252198    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s" event={"ID":"56402ba4-5419-40d2-b92e-2cf682432625","Type":"ContainerDied","Data":"8b4c6c976f4685b9ff61b64bca6a3aec6db61efb95f87054f1fdb0909c92a64c"}
Nov 03 17:51:59.252616 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:51:59.252255    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b4c6c976f4685b9ff61b64bca6a3aec6db61efb95f87054f1fdb0909c92a64c"
Nov 03 17:52:00.121524 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.121489    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"]
Nov 03 17:52:00.122299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.122277    4493 cpu_manager.go:414] "containerMap: removing container" podUID="82ec2497-20d5-42bd-ad51-827fd0eb926c" containerName="revoke-keys"
Nov 03 17:52:00.122299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.122299    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ec2497-20d5-42bd-ad51-827fd0eb926c" containerName="revoke-keys"
Nov 03 17:52:00.122506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.122361    4493 cpu_manager.go:414] "containerMap: removing container" podUID="56402ba4-5419-40d2-b92e-2cf682432625" containerName="revoke-keys"
Nov 03 17:52:00.122506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.122370    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56402ba4-5419-40d2-b92e-2cf682432625" containerName="revoke-keys"
Nov 03 17:52:00.122618 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.122601    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="56402ba4-5419-40d2-b92e-2cf682432625" containerName="revoke-keys"
Nov 03 17:52:00.122672 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.122638    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ec2497-20d5-42bd-ad51-827fd0eb926c" containerName="revoke-keys"
Nov 03 17:52:00.126432 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.126410    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.187627 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.187594    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"]
Nov 03 17:52:00.216217 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.216184    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.216387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.216297    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxd6\" (UniqueName: \"kubernetes.io/projected/ba4f867a-f1a7-4685-84c6-67ad1499ef31-kube-api-access-5sxd6\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.216387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.216336    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.231254 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.231218    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:52:00.231254 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.231233    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:52:00.231411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.231265    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:52:00.231411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.231325    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:52:00.317819 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.317788    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.318203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.317843    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxd6\" (UniqueName: \"kubernetes.io/projected/ba4f867a-f1a7-4685-84c6-67ad1499ef31-kube-api-access-5sxd6\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.318203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.317902    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.318593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.318570    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.320462 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.320440    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.381775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.381705    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxd6\" (UniqueName: \"kubernetes.io/projected/ba4f867a-f1a7-4685-84c6-67ad1499ef31-kube-api-access-5sxd6\") pod \"maas-api-revoke-sub-3c27b3b3-4zf6n\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.440083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.440042    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:00.824634 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:00.824611    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"]
Nov 03 17:52:00.826577 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:00.826549    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4f867a_f1a7_4685_84c6_67ad1499ef31.slice/crio-f8fdcd9eac513503d25ac203a1b6fa86b369695646371ed97c6490ea780200dc WatchSource:0}: Error finding container f8fdcd9eac513503d25ac203a1b6fa86b369695646371ed97c6490ea780200dc: Status 404 returned error can't find the container with id f8fdcd9eac513503d25ac203a1b6fa86b369695646371ed97c6490ea780200dc
Nov 03 17:52:01.266677 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:01.266639    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n" event={"ID":"ba4f867a-f1a7-4685-84c6-67ad1499ef31","Type":"ContainerStarted","Data":"7123628d53469e4234a55967ba5f788ec2ea1b0d724fe1bb2d8347ac43793c53"}
Nov 03 17:52:01.266897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:01.266698    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n" event={"ID":"ba4f867a-f1a7-4685-84c6-67ad1499ef31","Type":"ContainerStarted","Data":"f8fdcd9eac513503d25ac203a1b6fa86b369695646371ed97c6490ea780200dc"}
Nov 03 17:52:01.389142 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:01.389036    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n" podStartSLOduration=1.389022879 podStartE2EDuration="1.389022879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:01.386537085 +0000 UTC m=+1100.845708073" watchObservedRunningTime="2026-11-03 17:52:01.389022879 +0000 UTC m=+1100.848193865"
Nov 03 17:52:02.273086 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:02.273054    4493 generic.go:356] "Generic (PLEG): container finished" podID="ba4f867a-f1a7-4685-84c6-67ad1499ef31" containerID="7123628d53469e4234a55967ba5f788ec2ea1b0d724fe1bb2d8347ac43793c53" exitCode=0
Nov 03 17:52:02.273086 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:02.273091    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n" event={"ID":"ba4f867a-f1a7-4685-84c6-67ad1499ef31","Type":"ContainerDied","Data":"7123628d53469e4234a55967ba5f788ec2ea1b0d724fe1bb2d8347ac43793c53"}
Nov 03 17:52:03.211067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.211035    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"]
Nov 03 17:52:03.214845 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.214823    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.275236 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.275198    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"]
Nov 03 17:52:03.346533 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.346502    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcx4\" (UniqueName: \"kubernetes.io/projected/5cb97f7f-8c04-45b3-b97b-cf4607409e74-kube-api-access-qfcx4\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.346700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.346543    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.346700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.346642    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.411996 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.411969    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:03.448231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.448208    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcx4\" (UniqueName: \"kubernetes.io/projected/5cb97f7f-8c04-45b3-b97b-cf4607409e74-kube-api-access-qfcx4\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.448353 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.448250    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.448422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.448395    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.449048 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.448982    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.450700 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.450677    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.512128 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.512102    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcx4\" (UniqueName: \"kubernetes.io/projected/5cb97f7f-8c04-45b3-b97b-cf4607409e74-kube-api-access-qfcx4\") pod \"maas-api-revoke-sub-6e47aeb6-f8d6v\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.527963 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.527938    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:03.549712 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.549678    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4f867a-f1a7-4685-84c6-67ad1499ef31-kube-api-access-5sxd6\" (UniqueName: \"kubernetes.io/projected/ba4f867a-f1a7-4685-84c6-67ad1499ef31-kube-api-access-5sxd6\") pod \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") "
Nov 03 17:52:03.549854 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.549779    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-openshift-ca\") pod \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") "
Nov 03 17:52:03.549854 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.549828    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-xks-ca\") pod \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\" (UID: \"ba4f867a-f1a7-4685-84c6-67ad1499ef31\") "
Nov 03 17:52:03.550281 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.550251    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-openshift-ca" pod "ba4f867a-f1a7-4685-84c6-67ad1499ef31" (UID: "ba4f867a-f1a7-4685-84c6-67ad1499ef31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:03.552220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.552193    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4f867a-f1a7-4685-84c6-67ad1499ef31-kube-api-access-5sxd6" pod "ba4f867a-f1a7-4685-84c6-67ad1499ef31" (UID: "ba4f867a-f1a7-4685-84c6-67ad1499ef31"). InnerVolumeSpecName "kube-api-access-5sxd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:03.552585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.552570    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-xks-ca" pod "ba4f867a-f1a7-4685-84c6-67ad1499ef31" (UID: "ba4f867a-f1a7-4685-84c6-67ad1499ef31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:52:03.651571 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.651541    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxd6\" (UniqueName: \"kubernetes.io/projected/ba4f867a-f1a7-4685-84c6-67ad1499ef31-kube-api-access-5sxd6\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:03.651571 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.651569    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:03.651745 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.651579    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4f867a-f1a7-4685-84c6-67ad1499ef31-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:03.916372 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:03.916343    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"]
Nov 03 17:52:03.917403 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:03.917379    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb97f7f_8c04_45b3_b97b_cf4607409e74.slice/crio-239726d938b3215eab03c18ae4b0c979945d11300adeecc40017e2cd54da2a3a WatchSource:0}: Error finding container 239726d938b3215eab03c18ae4b0c979945d11300adeecc40017e2cd54da2a3a: Status 404 returned error can't find the container with id 239726d938b3215eab03c18ae4b0c979945d11300adeecc40017e2cd54da2a3a
Nov 03 17:52:04.286813 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.286779    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v" event={"ID":"5cb97f7f-8c04-45b3-b97b-cf4607409e74","Type":"ContainerStarted","Data":"b5b3d2e859e7fcfa7422ae246b7aec4809a7b8fe57c0e0ee2e025bb27fca4181"}
Nov 03 17:52:04.287272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.286833    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v" event={"ID":"5cb97f7f-8c04-45b3-b97b-cf4607409e74","Type":"ContainerStarted","Data":"239726d938b3215eab03c18ae4b0c979945d11300adeecc40017e2cd54da2a3a"}
Nov 03 17:52:04.288215 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.288198    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"
Nov 03 17:52:04.288282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.288239    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n" event={"ID":"ba4f867a-f1a7-4685-84c6-67ad1499ef31","Type":"ContainerDied","Data":"f8fdcd9eac513503d25ac203a1b6fa86b369695646371ed97c6490ea780200dc"}
Nov 03 17:52:04.288321 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.288282    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8fdcd9eac513503d25ac203a1b6fa86b369695646371ed97c6490ea780200dc"
Nov 03 17:52:04.408632 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.408574    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v" podStartSLOduration=1.408560595 podStartE2EDuration="1.408560595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:04.406583246 +0000 UTC m=+1103.865754232" watchObservedRunningTime="2026-11-03 17:52:04.408560595 +0000 UTC m=+1103.867731580"
Nov 03 17:52:04.786917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.786864    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"]
Nov 03 17:52:04.787320 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.787307    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ba4f867a-f1a7-4685-84c6-67ad1499ef31" containerName="revoke-keys"
Nov 03 17:52:04.787366 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.787322    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4f867a-f1a7-4685-84c6-67ad1499ef31" containerName="revoke-keys"
Nov 03 17:52:04.787420 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.787411    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4f867a-f1a7-4685-84c6-67ad1499ef31" containerName="revoke-keys"
Nov 03 17:52:04.790739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.790723    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.850127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.850094    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"]
Nov 03 17:52:04.864153 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.864119    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4sv\" (UniqueName: \"kubernetes.io/projected/777a3204-e98b-45c0-af1c-30b622f244ec-kube-api-access-kx4sv\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.864296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.864277    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.864447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.864428    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.965140 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.965108    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.965291 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.965234    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.965291 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.965274    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4sv\" (UniqueName: \"kubernetes.io/projected/777a3204-e98b-45c0-af1c-30b622f244ec-kube-api-access-kx4sv\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.965793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.965762    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:04.967762 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:04.967739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:05.029141 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:05.029114    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4sv\" (UniqueName: \"kubernetes.io/projected/777a3204-e98b-45c0-af1c-30b622f244ec-kube-api-access-kx4sv\") pod \"maas-api-revoke-sub-f082c984-stxff\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:05.103268 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:05.103197    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:05.295354 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:05.295319    4493 generic.go:356] "Generic (PLEG): container finished" podID="5cb97f7f-8c04-45b3-b97b-cf4607409e74" containerID="b5b3d2e859e7fcfa7422ae246b7aec4809a7b8fe57c0e0ee2e025bb27fca4181" exitCode=0
Nov 03 17:52:05.295735 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:05.295404    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v" event={"ID":"5cb97f7f-8c04-45b3-b97b-cf4607409e74","Type":"ContainerDied","Data":"b5b3d2e859e7fcfa7422ae246b7aec4809a7b8fe57c0e0ee2e025bb27fca4181"}
Nov 03 17:52:05.538354 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:05.538330    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"]
Nov 03 17:52:05.539255 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:05.539233    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777a3204_e98b_45c0_af1c_30b622f244ec.slice/crio-487985d731c1af3e04371144322bf9c49e777018c726311e8351aaba2302585c WatchSource:0}: Error finding container 487985d731c1af3e04371144322bf9c49e777018c726311e8351aaba2302585c: Status 404 returned error can't find the container with id 487985d731c1af3e04371144322bf9c49e777018c726311e8351aaba2302585c
Nov 03 17:52:06.301703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.301673    4493 generic.go:356] "Generic (PLEG): container finished" podID="777a3204-e98b-45c0-af1c-30b622f244ec" containerID="3862aa6cb1afeac2e3868aa071bf3d39ea21b2c7c79ba71887ca896e37637bba" exitCode=0
Nov 03 17:52:06.302101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.301746    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff" event={"ID":"777a3204-e98b-45c0-af1c-30b622f244ec","Type":"ContainerDied","Data":"3862aa6cb1afeac2e3868aa071bf3d39ea21b2c7c79ba71887ca896e37637bba"}
Nov 03 17:52:06.302101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.301802    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff" event={"ID":"777a3204-e98b-45c0-af1c-30b622f244ec","Type":"ContainerStarted","Data":"487985d731c1af3e04371144322bf9c49e777018c726311e8351aaba2302585c"}
Nov 03 17:52:06.449581 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.449557    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:06.582712 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.582637    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-xks-ca\") pod \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") "
Nov 03 17:52:06.582842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.582763    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb97f7f-8c04-45b3-b97b-cf4607409e74-kube-api-access-qfcx4\" (UniqueName: \"kubernetes.io/projected/5cb97f7f-8c04-45b3-b97b-cf4607409e74-kube-api-access-qfcx4\") pod \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") "
Nov 03 17:52:06.582842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.582803    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-openshift-ca\") pod \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\" (UID: \"5cb97f7f-8c04-45b3-b97b-cf4607409e74\") "
Nov 03 17:52:06.585965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.583691    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-openshift-ca" pod "5cb97f7f-8c04-45b3-b97b-cf4607409e74" (UID: "5cb97f7f-8c04-45b3-b97b-cf4607409e74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:06.594720 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.594684    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb97f7f-8c04-45b3-b97b-cf4607409e74-kube-api-access-qfcx4" pod "5cb97f7f-8c04-45b3-b97b-cf4607409e74" (UID: "5cb97f7f-8c04-45b3-b97b-cf4607409e74"). InnerVolumeSpecName "kube-api-access-qfcx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:06.595013 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.594764    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-xks-ca" pod "5cb97f7f-8c04-45b3-b97b-cf4607409e74" (UID: "5cb97f7f-8c04-45b3-b97b-cf4607409e74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:52:06.684967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.684928    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:06.684967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.684962    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcx4\" (UniqueName: \"kubernetes.io/projected/5cb97f7f-8c04-45b3-b97b-cf4607409e74-kube-api-access-qfcx4\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:06.684967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:06.684973    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb97f7f-8c04-45b3-b97b-cf4607409e74-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:06.992733 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:52:06.992691    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:52:07.314006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.313919    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"
Nov 03 17:52:07.314006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.313916    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v" event={"ID":"5cb97f7f-8c04-45b3-b97b-cf4607409e74","Type":"ContainerDied","Data":"239726d938b3215eab03c18ae4b0c979945d11300adeecc40017e2cd54da2a3a"}
Nov 03 17:52:07.314448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.314043    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="239726d938b3215eab03c18ae4b0c979945d11300adeecc40017e2cd54da2a3a"
Nov 03 17:52:07.452158 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.452131    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:07.593974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.593904    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777a3204-e98b-45c0-af1c-30b622f244ec-kube-api-access-kx4sv\" (UniqueName: \"kubernetes.io/projected/777a3204-e98b-45c0-af1c-30b622f244ec-kube-api-access-kx4sv\") pod \"777a3204-e98b-45c0-af1c-30b622f244ec\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") "
Nov 03 17:52:07.594142 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.593973    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-xks-ca\") pod \"777a3204-e98b-45c0-af1c-30b622f244ec\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") "
Nov 03 17:52:07.594142 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.594020    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-openshift-ca\") pod \"777a3204-e98b-45c0-af1c-30b622f244ec\" (UID: \"777a3204-e98b-45c0-af1c-30b622f244ec\") "
Nov 03 17:52:07.594433 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.594404    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-openshift-ca" pod "777a3204-e98b-45c0-af1c-30b622f244ec" (UID: "777a3204-e98b-45c0-af1c-30b622f244ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:07.596280 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.596250    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-xks-ca" pod "777a3204-e98b-45c0-af1c-30b622f244ec" (UID: "777a3204-e98b-45c0-af1c-30b622f244ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:52:07.596410 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.596384    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777a3204-e98b-45c0-af1c-30b622f244ec-kube-api-access-kx4sv" pod "777a3204-e98b-45c0-af1c-30b622f244ec" (UID: "777a3204-e98b-45c0-af1c-30b622f244ec"). InnerVolumeSpecName "kube-api-access-kx4sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:07.707679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.707641    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:07.707679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.707674    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4sv\" (UniqueName: \"kubernetes.io/projected/777a3204-e98b-45c0-af1c-30b622f244ec-kube-api-access-kx4sv\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:07.707679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:07.707688    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/777a3204-e98b-45c0-af1c-30b622f244ec-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:08.320969 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:08.320940    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"
Nov 03 17:52:08.321505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:08.320984    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff" event={"ID":"777a3204-e98b-45c0-af1c-30b622f244ec","Type":"ContainerDied","Data":"487985d731c1af3e04371144322bf9c49e777018c726311e8351aaba2302585c"}
Nov 03 17:52:08.321505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:08.321029    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="487985d731c1af3e04371144322bf9c49e777018c726311e8351aaba2302585c"
Nov 03 17:52:09.844769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.844735    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"]
Nov 03 17:52:09.845162 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.845147    4493 cpu_manager.go:414] "containerMap: removing container" podUID="777a3204-e98b-45c0-af1c-30b622f244ec" containerName="revoke-keys"
Nov 03 17:52:09.845220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.845164    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777a3204-e98b-45c0-af1c-30b622f244ec" containerName="revoke-keys"
Nov 03 17:52:09.845220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.845178    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5cb97f7f-8c04-45b3-b97b-cf4607409e74" containerName="revoke-keys"
Nov 03 17:52:09.845220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.845183    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb97f7f-8c04-45b3-b97b-cf4607409e74" containerName="revoke-keys"
Nov 03 17:52:09.845314 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.845248    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="777a3204-e98b-45c0-af1c-30b622f244ec" containerName="revoke-keys"
Nov 03 17:52:09.845314 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.845256    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb97f7f-8c04-45b3-b97b-cf4607409e74" containerName="revoke-keys"
Nov 03 17:52:09.850656 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.850625    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:09.929178 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.929136    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-tmp-dir\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:09.929375 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.929276    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-model-cache\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:09.929375 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.929353    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-home\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:09.929514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.929397    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f4752d-8f13-4214-83ea-1748907f4eb2-tls-certs\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:09.929576 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.929524    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn5ps\" (UniqueName: \"kubernetes.io/projected/d7f4752d-8f13-4214-83ea-1748907f4eb2-kube-api-access-cn5ps\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:09.929632 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.929614    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-dshm\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:09.955773 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.955740    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"distinct-w1-c76280-kserve-self-signed-certs\""
Nov 03 17:52:09.958705 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:09.958685    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"]
Nov 03 17:52:10.030549 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.030518    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn5ps\" (UniqueName: \"kubernetes.io/projected/d7f4752d-8f13-4214-83ea-1748907f4eb2-kube-api-access-cn5ps\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.030734 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.030588    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-dshm\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.030734 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.030611    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-tmp-dir\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.030734 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.030659    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-model-cache\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.030734 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.030675    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-home\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.030734 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.030701    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f4752d-8f13-4214-83ea-1748907f4eb2-tls-certs\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.031152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.031118    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-home\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.031152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.031142    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-model-cache\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.031309 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.031184    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-tmp-dir\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.033171 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.033146    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-dshm\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.033279 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.033261    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f4752d-8f13-4214-83ea-1748907f4eb2-tls-certs\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.094656 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.094628    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn5ps\" (UniqueName: \"kubernetes.io/projected/d7f4752d-8f13-4214-83ea-1748907f4eb2-kube-api-access-cn5ps\") pod \"distinct-w1-c76280-kserve-dddfbf958-lsbww\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.164537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.164433    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:10.571844 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:10.571819    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"]
Nov 03 17:52:10.572672 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:10.572652    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7f4752d_8f13_4214_83ea_1748907f4eb2.slice/crio-9dc9ef29e86e5da92a54f66c4f81db99b0dea80edc8c6f141e5f99b716825e99 WatchSource:0}: Error finding container 9dc9ef29e86e5da92a54f66c4f81db99b0dea80edc8c6f141e5f99b716825e99: Status 404 returned error can't find the container with id 9dc9ef29e86e5da92a54f66c4f81db99b0dea80edc8c6f141e5f99b716825e99
Nov 03 17:52:11.340818 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:11.340787    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" event={"ID":"d7f4752d-8f13-4214-83ea-1748907f4eb2","Type":"ContainerStarted","Data":"0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981"}
Nov 03 17:52:11.341319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:11.340840    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" event={"ID":"d7f4752d-8f13-4214-83ea-1748907f4eb2","Type":"ContainerStarted","Data":"9dc9ef29e86e5da92a54f66c4f81db99b0dea80edc8c6f141e5f99b716825e99"}
Nov 03 17:52:11.462982 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:11.462932    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podStartSLOduration=2.462918004 podStartE2EDuration="2.462918004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:11.461559347 +0000 UTC m=+1110.920730333" watchObservedRunningTime="2026-11-03 17:52:11.462918004 +0000 UTC m=+1110.922088987"
Nov 03 17:52:12.165350 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:12.165307    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:14.651315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.651277    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"]
Nov 03 17:52:14.655216 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.655196    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.715892 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.715845    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"]
Nov 03 17:52:14.762094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.762057    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:52:14.762094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.762083    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:52:14.762291 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.762110    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:52:14.762291 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.762178    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:52:14.780794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.780762    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.780794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.780796    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsz24\" (UniqueName: \"kubernetes.io/projected/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-kube-api-access-nsz24\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.781035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.780822    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.881610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.881578    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.881610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.881615    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsz24\" (UniqueName: \"kubernetes.io/projected/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-kube-api-access-nsz24\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.881861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.881639    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.882358 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.882338    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.884092 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.884074    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.946272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.946200    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsz24\" (UniqueName: \"kubernetes.io/projected/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-kube-api-access-nsz24\") pod \"maas-api-revoke-sub-acf0506f-pn7jb\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:14.969066 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:14.969032    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:15.350984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:15.350958    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"]
Nov 03 17:52:15.352307 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:15.352280    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8306dce6_39a4_4f03_bd6c_bdf46e84a8dc.slice/crio-1d5b1e712b62bbfc2430d747d2760c29520b7883002da01130306ff4ff8b40ac WatchSource:0}: Error finding container 1d5b1e712b62bbfc2430d747d2760c29520b7883002da01130306ff4ff8b40ac: Status 404 returned error can't find the container with id 1d5b1e712b62bbfc2430d747d2760c29520b7883002da01130306ff4ff8b40ac
Nov 03 17:52:15.363605 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:15.363577    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb" event={"ID":"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc","Type":"ContainerStarted","Data":"1d5b1e712b62bbfc2430d747d2760c29520b7883002da01130306ff4ff8b40ac"}
Nov 03 17:52:16.370777 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:16.370696    4493 generic.go:356] "Generic (PLEG): container finished" podID="8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" containerID="be2d12b5a247cae046e24ed11ae0866be42b65c2148801df99eb90d9e6aecd62" exitCode=0
Nov 03 17:52:16.371145 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:16.370781    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb" event={"ID":"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc","Type":"ContainerDied","Data":"be2d12b5a247cae046e24ed11ae0866be42b65c2148801df99eb90d9e6aecd62"}
Nov 03 17:52:17.524345 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.524308    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:17.710327 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.710254    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-kube-api-access-nsz24\" (UniqueName: \"kubernetes.io/projected/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-kube-api-access-nsz24\") pod \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") "
Nov 03 17:52:17.710327 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.710306    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-xks-ca\") pod \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") "
Nov 03 17:52:17.710488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.710372    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-openshift-ca\") pod \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\" (UID: \"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc\") "
Nov 03 17:52:17.710713 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.710690    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-openshift-ca" pod "8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" (UID: "8306dce6-39a4-4f03-bd6c-bdf46e84a8dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:17.712495 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.712473    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-xks-ca" pod "8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" (UID: "8306dce6-39a4-4f03-bd6c-bdf46e84a8dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:52:17.712556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.712538    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-kube-api-access-nsz24" pod "8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" (UID: "8306dce6-39a4-4f03-bd6c-bdf46e84a8dc"). InnerVolumeSpecName "kube-api-access-nsz24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:17.811665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.811632    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:17.811665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.811659    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsz24\" (UniqueName: \"kubernetes.io/projected/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-kube-api-access-nsz24\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:17.811665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:17.811672    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:18.385252 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:18.385222    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"
Nov 03 17:52:18.385252 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:18.385237    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb" event={"ID":"8306dce6-39a4-4f03-bd6c-bdf46e84a8dc","Type":"ContainerDied","Data":"1d5b1e712b62bbfc2430d747d2760c29520b7883002da01130306ff4ff8b40ac"}
Nov 03 17:52:18.385514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:18.385281    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5b1e712b62bbfc2430d747d2760c29520b7883002da01130306ff4ff8b40ac"
Nov 03 17:52:20.165098 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:20.165061    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:20.178943 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:20.178912    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:20.407934 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:20.407907    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:52:21.470120 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.470044    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"]
Nov 03 17:52:21.470494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.470475    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" containerName="revoke-keys"
Nov 03 17:52:21.470494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.470488    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" containerName="revoke-keys"
Nov 03 17:52:21.470595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.470584    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" containerName="revoke-keys"
Nov 03 17:52:21.473967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.473951    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.530093 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.530061    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-776ba7-openshift-default-dockercfg-nv8qc\""
Nov 03 17:52:21.549738 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.549699    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.549887 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.549763    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.549927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.549864    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.549972 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.549957    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.550011 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.549982    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.550105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.550093    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwc78\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-kube-api-access-gwc78\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.550150 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.550140    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.550202 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.550185    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.550236 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.550217    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.584067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.584032    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"]
Nov 03 17:52:21.651196 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651158    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwc78\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-kube-api-access-gwc78\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651220    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651242    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651261    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651278    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651308    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651337    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651669 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651382    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651669 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651411    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651719    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651760    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.651931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651909    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.652001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.651938    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.652141 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.652121    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.653772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.653748    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.654014 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.653995    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.715707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.715670    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.715903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.715814    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwc78\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-kube-api-access-gwc78\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.783627 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.783551    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"]
Nov 03 17:52:21.786587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.786557    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:21.788374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.788356    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.845076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.845049    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:52:21.845214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.845142    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:52:21.845214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.845173    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:52:21.845214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.845190    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:52:21.854925 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.849183    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"]
Nov 03 17:52:21.854925 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.853248    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crlsx\" (UniqueName: \"kubernetes.io/projected/a4224491-4ba8-40d6-894f-b5c381cae384-kube-api-access-crlsx\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.854925 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.853295    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.854925 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.853392    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.954405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.954328    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crlsx\" (UniqueName: \"kubernetes.io/projected/a4224491-4ba8-40d6-894f-b5c381cae384-kube-api-access-crlsx\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.954405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.954388    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.954644 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.954480    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.955097 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.955078    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:21.957054 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:21.957025    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:22.018623 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.018589    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crlsx\" (UniqueName: \"kubernetes.io/projected/a4224491-4ba8-40d6-894f-b5c381cae384-kube-api-access-crlsx\") pod \"maas-api-revoke-sub-a35cf42d-2z5mc\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:22.120409 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.120321    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:22.224728 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:22.224696    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd43a67_2eea_4b95_8388_5a06f65d4ca4.slice/crio-e80ed6587e298befdb21e858d92a1831219646e372db64b56cb61cbebaac0f0f WatchSource:0}: Error finding container e80ed6587e298befdb21e858d92a1831219646e372db64b56cb61cbebaac0f0f: Status 404 returned error can't find the container with id e80ed6587e298befdb21e858d92a1831219646e372db64b56cb61cbebaac0f0f
Nov 03 17:52:22.227144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.227108    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:52:22.227238 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.227180    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:52:22.227238 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.227210    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:52:22.227402 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.227380    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"]
Nov 03 17:52:22.408732 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.408640    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" event={"ID":"0cd43a67-2eea-4b95-8388-5a06f65d4ca4","Type":"ContainerStarted","Data":"53999f6cb2dd3b1ffd55321e0a172f84302d2ba773b368a9cc951047f98d6590"}
Nov 03 17:52:22.408732 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.408707    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" event={"ID":"0cd43a67-2eea-4b95-8388-5a06f65d4ca4","Type":"ContainerStarted","Data":"e80ed6587e298befdb21e858d92a1831219646e372db64b56cb61cbebaac0f0f"}
Nov 03 17:52:22.502083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.502053    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"]
Nov 03 17:52:22.502968 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:22.502940    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4224491_4ba8_40d6_894f_b5c381cae384.slice/crio-d35696153f4e6f6d7deb9e637ed2a033c8abc0195429c403d7c8c9b88dc0e795 WatchSource:0}: Error finding container d35696153f4e6f6d7deb9e637ed2a033c8abc0195429c403d7c8c9b88dc0e795: Status 404 returned error can't find the container with id d35696153f4e6f6d7deb9e637ed2a033c8abc0195429c403d7c8c9b88dc0e795
Nov 03 17:52:22.534305 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.534254    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podStartSLOduration=1.534239175 podStartE2EDuration="1.534239175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:22.533591556 +0000 UTC m=+1121.992762566" watchObservedRunningTime="2026-11-03 17:52:22.534239175 +0000 UTC m=+1121.993410161"
Nov 03 17:52:22.787538 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:22.787506    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:23.445792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:23.445592    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc" event={"ID":"a4224491-4ba8-40d6-894f-b5c381cae384","Type":"ContainerStarted","Data":"e745926234ae02e5b80b5b1f0c926882c1f5c90bc1cab9da734fafcb48eceeec"}
Nov 03 17:52:23.445792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:23.445667    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc" event={"ID":"a4224491-4ba8-40d6-894f-b5c381cae384","Type":"ContainerStarted","Data":"d35696153f4e6f6d7deb9e637ed2a033c8abc0195429c403d7c8c9b88dc0e795"}
Nov 03 17:52:23.565739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:23.565689    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc" podStartSLOduration=2.565674477 podStartE2EDuration="2.565674477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:23.565003028 +0000 UTC m=+1123.024174016" watchObservedRunningTime="2026-11-03 17:52:23.565674477 +0000 UTC m=+1123.024845500"
Nov 03 17:52:23.789119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:23.789082    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:52:23.789298 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:23.789139    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:52:24.452538 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:24.452503    4493 generic.go:356] "Generic (PLEG): container finished" podID="a4224491-4ba8-40d6-894f-b5c381cae384" containerID="e745926234ae02e5b80b5b1f0c926882c1f5c90bc1cab9da734fafcb48eceeec" exitCode=0
Nov 03 17:52:24.452731 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:24.452596    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc" event={"ID":"a4224491-4ba8-40d6-894f-b5c381cae384","Type":"ContainerDied","Data":"e745926234ae02e5b80b5b1f0c926882c1f5c90bc1cab9da734fafcb48eceeec"}
Nov 03 17:52:24.787781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:24.787746    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:52:24.788286 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:24.787809    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:52:25.643733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.643701    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:25.693253 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.693223    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4224491-4ba8-40d6-894f-b5c381cae384-kube-api-access-crlsx\" (UniqueName: \"kubernetes.io/projected/a4224491-4ba8-40d6-894f-b5c381cae384-kube-api-access-crlsx\") pod \"a4224491-4ba8-40d6-894f-b5c381cae384\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") "
Nov 03 17:52:25.693486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.693269    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-openshift-ca\") pod \"a4224491-4ba8-40d6-894f-b5c381cae384\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") "
Nov 03 17:52:25.693486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.693381    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-xks-ca\") pod \"a4224491-4ba8-40d6-894f-b5c381cae384\" (UID: \"a4224491-4ba8-40d6-894f-b5c381cae384\") "
Nov 03 17:52:25.693710 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.693686    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-openshift-ca" pod "a4224491-4ba8-40d6-894f-b5c381cae384" (UID: "a4224491-4ba8-40d6-894f-b5c381cae384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:25.695333 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.695303    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4224491-4ba8-40d6-894f-b5c381cae384-kube-api-access-crlsx" pod "a4224491-4ba8-40d6-894f-b5c381cae384" (UID: "a4224491-4ba8-40d6-894f-b5c381cae384"). InnerVolumeSpecName "kube-api-access-crlsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:25.695689 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.695665    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-xks-ca" pod "a4224491-4ba8-40d6-894f-b5c381cae384" (UID: "a4224491-4ba8-40d6-894f-b5c381cae384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:52:25.787423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.787344    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:52:25.787423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.787401    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:52:25.794975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.794934    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crlsx\" (UniqueName: \"kubernetes.io/projected/a4224491-4ba8-40d6-894f-b5c381cae384-kube-api-access-crlsx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:25.794975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.794961    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:25.794975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:25.794972    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4224491-4ba8-40d6-894f-b5c381cae384-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:26.473003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:26.472964    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc" event={"ID":"a4224491-4ba8-40d6-894f-b5c381cae384","Type":"ContainerDied","Data":"d35696153f4e6f6d7deb9e637ed2a033c8abc0195429c403d7c8c9b88dc0e795"}
Nov 03 17:52:26.473003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:26.472993    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"
Nov 03 17:52:26.473003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:26.473013    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35696153f4e6f6d7deb9e637ed2a033c8abc0195429c403d7c8c9b88dc0e795"
Nov 03 17:52:26.786981 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:26.786896    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:52:26.786981 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:26.786947    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:52:27.787541 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:27.787505    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:52:27.788119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:27.787584    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:52:28.787023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:28.786980    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 17:52:28.787227 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:28.787033    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 17:52:29.791272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:29.791232    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:29.791680 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:29.791660    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:29.792368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:29.792345    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 17:52:36.776672 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:36.776629    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"]
Nov 03 17:52:36.777179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:36.776855    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy" containerID="cri-o://4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8" gracePeriod=30
Nov 03 17:52:36.995810 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:52:36.995771    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:52:38.854339 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.854305    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"]
Nov 03 17:52:38.854733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.854717    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a4224491-4ba8-40d6-894f-b5c381cae384" containerName="revoke-keys"
Nov 03 17:52:38.854733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.854728    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4224491-4ba8-40d6-894f-b5c381cae384" containerName="revoke-keys"
Nov 03 17:52:38.854820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.854810    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4224491-4ba8-40d6-894f-b5c381cae384" containerName="revoke-keys"
Nov 03 17:52:38.858396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.858381    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.914044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.914021    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-548ca1-openshift-default-dockercfg-z6wdl\""
Nov 03 17:52:38.966753 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.966715    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"]
Nov 03 17:52:38.967361 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967339    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6v8n\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-kube-api-access-l6v8n\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967470 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967411    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967544 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967517    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967573    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967620 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967605    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c9fa4-86ff-4958-b699-1156e7df1a3a-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967653 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967641    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967673    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967718 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967702    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:38.967779 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:38.967753    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.069271 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069235    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6v8n\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-kube-api-access-l6v8n\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.069454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069342    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.069454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069385    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.069454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069426    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.069641 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069461    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c9fa4-86ff-4958-b699-1156e7df1a3a-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.069641 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069507    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.069641 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069539    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.070003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069571    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.070108 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069822    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.070108 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069901    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.070108 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.070067    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.070108 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.069973    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.070341 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.070323    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.070488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.070455    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c9fa4-86ff-4958-b699-1156e7df1a3a-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.072039 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.072018    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.072376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.072358    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.138493 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.138431    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.138655 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.138637    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6v8n\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-kube-api-access-l6v8n\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.171428 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.171397    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:39.672224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.672193    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"]
Nov 03 17:52:39.672354 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:52:39.672322    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049c9fa4_86ff_4958_b699_1156e7df1a3a.slice/crio-349056785b4e26a08c5c42f3bb0c1fdde8d5979dc3b33592b61020a144daaa7c WatchSource:0}: Error finding container 349056785b4e26a08c5c42f3bb0c1fdde8d5979dc3b33592b61020a144daaa7c: Status 404 returned error can't find the container with id 349056785b4e26a08c5c42f3bb0c1fdde8d5979dc3b33592b61020a144daaa7c
Nov 03 17:52:39.674701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.674672    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:52:39.674768 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.674733    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:52:39.674803 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:39.674765    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:52:40.554524 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:40.554487    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" event={"ID":"049c9fa4-86ff-4958-b699-1156e7df1a3a","Type":"ContainerStarted","Data":"ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567"}
Nov 03 17:52:40.554942 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:40.554573    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" event={"ID":"049c9fa4-86ff-4958-b699-1156e7df1a3a","Type":"ContainerStarted","Data":"349056785b4e26a08c5c42f3bb0c1fdde8d5979dc3b33592b61020a144daaa7c"}
Nov 03 17:52:40.782781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:40.782730    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podStartSLOduration=2.782716614 podStartE2EDuration="2.782716614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:40.730803652 +0000 UTC m=+1140.189974646" watchObservedRunningTime="2026-11-03 17:52:40.782716614 +0000 UTC m=+1140.241887599"
Nov 03 17:52:41.171665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:41.171621    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:41.173897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:41.173850    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 17:52:41.174058 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:41.173944    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 17:52:42.073461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.073425    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:52:42.107813 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.107789    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-envoy\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.107976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.107820    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-certs\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.107976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.107843    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93dc0e29-5de3-43f6-a09a-e46774183e46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc0e29-5de3-43f6-a09a-e46774183e46-istiod-ca-cert\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.107976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.107939    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-podinfo\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.107976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.107958    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-token\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-token\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.108191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.107995    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-data\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.108191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108026    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-socket\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.108191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108104    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-kube-api-access-sgqp2\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-kube-api-access-sgqp2\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.108191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108111    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-certs" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:42.108191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108153    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-credential-socket\") pod \"93dc0e29-5de3-43f6-a09a-e46774183e46\" (UID: \"93dc0e29-5de3-43f6-a09a-e46774183e46\") "
Nov 03 17:52:42.108451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108224    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-data" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:42.108451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108356    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-socket" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:42.108451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108396    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93dc0e29-5de3-43f6-a09a-e46774183e46-istiod-ca-cert" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:52:42.108687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108585    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-credential-socket" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:42.108687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108646    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.108687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108662    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.108687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108671    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.108687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.108681    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc0e29-5de3-43f6-a09a-e46774183e46-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.110363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.110334    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-envoy" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 17:52:42.110363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.110349    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-podinfo" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 17:52:42.110529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.110406    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-token" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:42.110529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.110514    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-kube-api-access-sgqp2" pod "93dc0e29-5de3-43f6-a09a-e46774183e46" (UID: "93dc0e29-5de3-43f6-a09a-e46774183e46"). InnerVolumeSpecName "kube-api-access-sgqp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:52:42.172240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.172211    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 17:52:42.172380 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.172269    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 17:52:42.209973 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.209945    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.209973 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.209970    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.210144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.209984    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgqp2\" (UniqueName: \"kubernetes.io/projected/93dc0e29-5de3-43f6-a09a-e46774183e46-kube-api-access-sgqp2\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.210144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.209998    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.210144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.210009    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc0e29-5de3-43f6-a09a-e46774183e46-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:52:42.574895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.574829    4493 generic.go:356] "Generic (PLEG): container finished" podID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerID="4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8" exitCode=0
Nov 03 17:52:42.575059 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.574974    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"
Nov 03 17:52:42.575059 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.575002    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" event={"ID":"93dc0e29-5de3-43f6-a09a-e46774183e46","Type":"ContainerDied","Data":"4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8"}
Nov 03 17:52:42.575136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.575066    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb" event={"ID":"93dc0e29-5de3-43f6-a09a-e46774183e46","Type":"ContainerDied","Data":"55528ce7674565406121d8cac384f1a80158adb66234b7041af414046039be62"}
Nov 03 17:52:42.575136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.575094    4493 scope.go:122] "RemoveContainer" containerID="4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8"
Nov 03 17:52:42.587145 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.587030    4493 scope.go:122] "RemoveContainer" containerID="4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8"
Nov 03 17:52:42.587354 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:52:42.587329    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8\": container with ID starting with 4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8 not found: ID does not exist" containerID="4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8"
Nov 03 17:52:42.587437 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.587362    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8"} err="failed to get container status \"4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8\": rpc error: code = NotFound desc = could not find container \"4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8\": container with ID starting with 4dbf6bca75966f9d7fb4efed4cb3900845857ec906a4d4cdf999d13d97f34fe8 not found: ID does not exist"
Nov 03 17:52:42.761681 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.761414    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"]
Nov 03 17:52:42.809815 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:42.809783    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-29cqb"]
Nov 03 17:52:43.172584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:43.172555    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 17:52:43.172979 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:43.172601    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 17:52:44.171984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:44.171924    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 17:52:44.172158 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:44.172020    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 17:52:44.724584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:44.724149    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" path="/var/lib/kubelet/pods/93dc0e29-5de3-43f6-a09a-e46774183e46/volumes"
Nov 03 17:52:45.176974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:45.176441    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 17:52:45.176974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:45.176550    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 17:52:46.172735 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:46.172700    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 17:52:46.173282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:46.172759    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 17:52:47.177022 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:47.176992    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:47.177405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:47.177384    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:52:47.178115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:52:47.178097    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 17:53:06.992371 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:53:06.992267    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:53:14.060105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.060068    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"]
Nov 03 17:53:14.060524 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.060467    4493 cpu_manager.go:414] "containerMap: removing container" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy"
Nov 03 17:53:14.060524 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.060477    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy"
Nov 03 17:53:14.060600 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.060570    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="93dc0e29-5de3-43f6-a09a-e46774183e46" containerName="istio-proxy"
Nov 03 17:53:14.064084 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.064068    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.099041 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.099010    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2x9r\" (UniqueName: \"kubernetes.io/projected/383193f6-170f-4f03-a9f8-4fd37e2e11e1-kube-api-access-b2x9r\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.099224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.099053    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-dshm\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.099224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.099079    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tmp-dir\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.099224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.099206    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-model-cache\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.099399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.099269    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tls-certs\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.099399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.099296    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-home\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.177762 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.177730    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"distinct-w3-776ba7-kserve-self-signed-certs\""
Nov 03 17:53:14.180910 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.180887    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"]
Nov 03 17:53:14.200467 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.200432    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-model-cache\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.200629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.200516    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tls-certs\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.200629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.200548    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-home\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.200629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.200621    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2x9r\" (UniqueName: \"kubernetes.io/projected/383193f6-170f-4f03-a9f8-4fd37e2e11e1-kube-api-access-b2x9r\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.200811 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.200667    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-dshm\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.200811 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.200702    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tmp-dir\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.200944 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.200926    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-model-cache\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.201046 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.201024    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tmp-dir\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.201591 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.201136    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-home\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.204263 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.204230    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-dshm\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.204636 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.204601    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tls-certs\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.266229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.266194    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2x9r\" (UniqueName: \"kubernetes.io/projected/383193f6-170f-4f03-a9f8-4fd37e2e11e1-kube-api-access-b2x9r\") pod \"distinct-w3-776ba7-kserve-9b6b5b568-nxhpt\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.376017 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.375919    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:14.763363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:14.763331    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"]
Nov 03 17:53:14.763567 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:53:14.763541    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod383193f6_170f_4f03_a9f8_4fd37e2e11e1.slice/crio-e04260554dfb8af1f87f488926529a084930d1c315aec5606e74edb47e18eed6 WatchSource:0}: Error finding container e04260554dfb8af1f87f488926529a084930d1c315aec5606e74edb47e18eed6: Status 404 returned error can't find the container with id e04260554dfb8af1f87f488926529a084930d1c315aec5606e74edb47e18eed6
Nov 03 17:53:15.771758 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:15.771724    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" event={"ID":"383193f6-170f-4f03-a9f8-4fd37e2e11e1","Type":"ContainerStarted","Data":"92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733"}
Nov 03 17:53:15.772218 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:15.771771    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" event={"ID":"383193f6-170f-4f03-a9f8-4fd37e2e11e1","Type":"ContainerStarted","Data":"e04260554dfb8af1f87f488926529a084930d1c315aec5606e74edb47e18eed6"}
Nov 03 17:53:15.897895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:15.897827    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podStartSLOduration=2.8978137349999997 podStartE2EDuration="2.897813735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:15.895511227 +0000 UTC m=+1175.354682212" watchObservedRunningTime="2026-11-03 17:53:15.897813735 +0000 UTC m=+1175.356984720"
Nov 03 17:53:16.376065 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:16.376029    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:24.376963 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:24.376928    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:24.389684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:24.389659    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:24.830588 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:24.830561    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:53:36.745681 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.745648    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"]
Nov 03 17:53:36.749389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.749371    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.805148 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.805124    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f7d463-openshift-default-dockercfg-wqnxq\""
Nov 03 17:53:36.829554 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829520    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829673 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829568    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829725 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829668    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls4x9\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-kube-api-access-ls4x9\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829725    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829835 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829816    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829848    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829867    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829949    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.829994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.829978    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.858780 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.858750    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"]
Nov 03 17:53:36.931503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931475    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls4x9\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-kube-api-access-ls4x9\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931654 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931526    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931654 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931587    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931654 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931617    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931654 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931639    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931672    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931719    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931793    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.931866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931831    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.932099 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.931959    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.932195 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.932171    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.932243 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.932188    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.932294 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.932245    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.932421 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.932394    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.934035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.934013    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.934361 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.934340    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.991607 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:53:36.991574    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:53:36.996233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.996173    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:36.996338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:36.996296    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls4x9\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-kube-api-access-ls4x9\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:37.062426 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:37.062400    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:37.502457 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:53:37.502427    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63c2ba1_ba7a_4fcd_9624_a929d97a50e3.slice/crio-836dcae4c26d7ecb8bb42146b6821747ae6747dd2f0f4122ca8a5b229facf89b WatchSource:0}: Error finding container 836dcae4c26d7ecb8bb42146b6821747ae6747dd2f0f4122ca8a5b229facf89b: Status 404 returned error can't find the container with id 836dcae4c26d7ecb8bb42146b6821747ae6747dd2f0f4122ca8a5b229facf89b
Nov 03 17:53:37.502739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:37.502714    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"]
Nov 03 17:53:37.505078 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:37.505050    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:37.505143 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:37.505121    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:37.505184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:37.505149    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:37.895992 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:37.895905    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" event={"ID":"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3","Type":"ContainerStarted","Data":"8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a"}
Nov 03 17:53:37.896449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:37.895995    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" event={"ID":"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3","Type":"ContainerStarted","Data":"836dcae4c26d7ecb8bb42146b6821747ae6747dd2f0f4122ca8a5b229facf89b"}
Nov 03 17:53:38.024929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:38.024856    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podStartSLOduration=2.024842296 podStartE2EDuration="2.024842296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:38.022585484 +0000 UTC m=+1197.481756471" watchObservedRunningTime="2026-11-03 17:53:38.024842296 +0000 UTC m=+1197.484013283"
Nov 03 17:53:38.063992 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:38.063632    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:38.065672 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:38.065640    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:53:38.065841 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:38.065717    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:53:39.063523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:39.063482    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:53:39.064023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:39.063544    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:53:40.062891 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:40.062833    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:53:40.063068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:40.062925    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:53:41.062818 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:41.062784    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:53:41.063265 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:41.062846    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:53:42.063929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:42.063858    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:53:42.064324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:42.063948    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:53:43.063659 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:43.063620    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:53:43.063835 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:43.063675    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:53:44.063394 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:44.063358    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 17:53:44.063792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:44.063414    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 17:53:45.067390 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:45.067358    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:45.067850 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:45.067826    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:45.068462 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:45.068441    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 17:53:51.738096 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.738012    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"]
Nov 03 17:53:51.743600 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.743581    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.804241 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.804208    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"]
Nov 03 17:53:51.849991 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.849962    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-07f297-openshift-default-dockercfg-dfdjr\""
Nov 03 17:53:51.902955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.902918    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a008924-c63c-4bea-9dcf-438e683276be-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903146 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.902964    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903146 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.902991    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903271 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.903182    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a008924-c63c-4bea-9dcf-438e683276be-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903327 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.903291    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.903432    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903533 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.903514    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxvqf\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-kube-api-access-sxvqf\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.903585    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:51.903654 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:51.903614    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.004465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004376    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a008924-c63c-4bea-9dcf-438e683276be-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.004465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004444    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.004703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004506    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.004703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004540    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxvqf\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-kube-api-access-sxvqf\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.004703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004583    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.004887 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004678    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.004951 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004912    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a008924-c63c-4bea-9dcf-438e683276be-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.005014 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.004965    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.005106 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.005074    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.005278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.005256    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a008924-c63c-4bea-9dcf-438e683276be-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.005359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.005273    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.005359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.005288    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.005359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.005312    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.005614 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.005591    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.007092 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.007071    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.007686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.007663    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a008924-c63c-4bea-9dcf-438e683276be-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.069623 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.069597    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxvqf\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-kube-api-access-sxvqf\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.069765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.069637    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.356733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.356647    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:52.797684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.797661    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"]
Nov 03 17:53:52.798747 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:53:52.798725    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a008924_c63c_4bea_9dcf_438e683276be.slice/crio-5ccf3dd8f15066076e95ab9bb3a937f2dfc1faedee9279c5ddbb90b3b1aa5964 WatchSource:0}: Error finding container 5ccf3dd8f15066076e95ab9bb3a937f2dfc1faedee9279c5ddbb90b3b1aa5964: Status 404 returned error can't find the container with id 5ccf3dd8f15066076e95ab9bb3a937f2dfc1faedee9279c5ddbb90b3b1aa5964
Nov 03 17:53:52.801184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.801145    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:52.801305 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.801232    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:52.801305 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:52.801290    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:53:53.013994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:53.013955    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" event={"ID":"8a008924-c63c-4bea-9dcf-438e683276be","Type":"ContainerStarted","Data":"38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6"}
Nov 03 17:53:53.014187 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:53.014024    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" event={"ID":"8a008924-c63c-4bea-9dcf-438e683276be","Type":"ContainerStarted","Data":"5ccf3dd8f15066076e95ab9bb3a937f2dfc1faedee9279c5ddbb90b3b1aa5964"}
Nov 03 17:53:53.143025 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:53.142917    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podStartSLOduration=2.1429026 podStartE2EDuration="2.1429026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:53.139261266 +0000 UTC m=+1212.598432289" watchObservedRunningTime="2026-11-03 17:53:53.1429026 +0000 UTC m=+1212.602073586"
Nov 03 17:53:53.357119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:53.357084    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:53.358906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:53.358865    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:53.359033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:53.358931    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 17:53:54.357329 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:54.357292    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:54.357707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:54.357346    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 17:53:55.357289 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:55.357253    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:55.357516 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:55.357306    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 17:53:56.357661 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:56.357632    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:56.358028 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:56.357690    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 17:53:57.357484 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:57.357453    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:57.357749 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:57.357520    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 17:53:58.357269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:58.357234    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 17:53:58.357453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:58.357299    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 17:53:59.678115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:59.678079    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:59.678803 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:59.678785    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:53:59.679895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:53:59.679850    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 17:54:06.992317 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:54:06.992278    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:54:36.992512 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:54:36.992477    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:54:53.054427 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.054394    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"]
Nov 03 17:54:53.058295 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.058274    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.113460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.113431    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c76280-openshift-default-dockercfg-mtxr6\""
Nov 03 17:54:53.160111 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160082    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160268 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160139    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160339 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160271    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160339 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160324    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160412    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6qq\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-kube-api-access-kp6qq\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160487    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160541 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160511    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160597 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160545    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.160645 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.160619    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.166439 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.166412    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"]
Nov 03 17:54:53.261247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261213    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261416 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261258    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261416 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261307    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6qq\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-kube-api-access-kp6qq\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261416 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261373    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261416 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261396    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261448    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261480    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261535    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261586    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261834 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261693    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.261966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.261945    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.262129 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.262108    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.262221 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.262077    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.262332 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.262188    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.263687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.263667    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.264223 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.264205    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.326237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.326174    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.326348 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.326275    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6qq\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-kube-api-access-kp6qq\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.371383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.371353    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:53.808950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.808927    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"]
Nov 03 17:54:53.810993 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:54:53.810968    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d84132_5d5f_4e2e_92f6_6ac5cb8f8819.slice/crio-a17e42379afc1d9020289617b739375647fd8e30e3ddee139945944ecd5399ab WatchSource:0}: Error finding container a17e42379afc1d9020289617b739375647fd8e30e3ddee139945944ecd5399ab: Status 404 returned error can't find the container with id a17e42379afc1d9020289617b739375647fd8e30e3ddee139945944ecd5399ab
Nov 03 17:54:53.813255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.813222    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:54:53.813314 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.813292    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:54:53.813350 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:53.813321    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:54:54.400149 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:54.400107    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" event={"ID":"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819","Type":"ContainerStarted","Data":"139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470"}
Nov 03 17:54:54.400637 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:54.400174    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" event={"ID":"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819","Type":"ContainerStarted","Data":"a17e42379afc1d9020289617b739375647fd8e30e3ddee139945944ecd5399ab"}
Nov 03 17:54:55.371548 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:55.371515    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:54:55.372996 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:55.372968    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:55.373130 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:55.373016    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 17:54:56.371926 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:56.371895    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:56.372300 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:56.371957    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 17:54:57.371884 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:57.371840    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:57.372065 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:57.371953    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 17:54:58.371803 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:58.371770    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:58.372003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:58.371833    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 17:54:59.371909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:59.371854    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:54:59.372318 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:54:59.371919    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 17:55:00.372635 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:00.372600    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 17:55:00.373124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:00.372665    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 17:55:01.376133 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:01.376101    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:55:01.376529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:01.376482    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:55:01.377214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:01.377199    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 17:55:01.554707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:01.554654    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podStartSLOduration=8.554641268 podStartE2EDuration="8.554641268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:54.541244086 +0000 UTC m=+1274.000415073" watchObservedRunningTime="2026-11-03 17:55:01.554641268 +0000 UTC m=+1281.013812335"
Nov 03 17:55:06.994694 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:55:06.994659    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:55:21.075094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:21.075014    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"]
Nov 03 17:55:21.125336 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:21.125304    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394bd57-jthjz"]
Nov 03 17:55:22.712652 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:22.712622    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce61bc3-ad99-4b69-a32a-afd8ffe2e728" path="/var/lib/kubelet/pods/fce61bc3-ad99-4b69-a32a-afd8ffe2e728/volumes"
Nov 03 17:55:25.741053 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.741016    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"]
Nov 03 17:55:25.745770 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.745750    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.805311 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.805279    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"]
Nov 03 17:55:25.842981 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.842943    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.843155 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.843081    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4j9r\" (UniqueName: \"kubernetes.io/projected/f3201103-f5be-48f9-95fa-e61ed2602ad5-kube-api-access-n4j9r\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.843201 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.843165    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.851665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.851639    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:55:25.851775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.851661    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:55:25.851775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.851661    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:55:25.851775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.851722    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:55:25.944828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.944790    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.945045 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.944931    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4j9r\" (UniqueName: \"kubernetes.io/projected/f3201103-f5be-48f9-95fa-e61ed2602ad5-kube-api-access-n4j9r\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.945045 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.945012    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.945540 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.945521    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:25.947456 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:25.947439    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:26.012247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:26.012174    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4j9r\" (UniqueName: \"kubernetes.io/projected/f3201103-f5be-48f9-95fa-e61ed2602ad5-kube-api-access-n4j9r\") pod \"maas-api-revoke-sub-f0831a2b-hd8dd\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:26.059150 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:26.059121    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:26.444759 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:55:26.444726    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3201103_f5be_48f9_95fa_e61ed2602ad5.slice/crio-f0d65c6aa241d4ef1fd0fd3283099f0399c24cd726c6bd3c44097f1f459acb0a WatchSource:0}: Error finding container f0d65c6aa241d4ef1fd0fd3283099f0399c24cd726c6bd3c44097f1f459acb0a: Status 404 returned error can't find the container with id f0d65c6aa241d4ef1fd0fd3283099f0399c24cd726c6bd3c44097f1f459acb0a
Nov 03 17:55:26.445323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:26.445300    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"]
Nov 03 17:55:26.626666 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:26.626632    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd" event={"ID":"f3201103-f5be-48f9-95fa-e61ed2602ad5","Type":"ContainerStarted","Data":"f0d65c6aa241d4ef1fd0fd3283099f0399c24cd726c6bd3c44097f1f459acb0a"}
Nov 03 17:55:27.634547 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:27.634506    4493 generic.go:356] "Generic (PLEG): container finished" podID="f3201103-f5be-48f9-95fa-e61ed2602ad5" containerID="ceae5a62a99192cdcdf2fae8ea86758b348dae59338ef31783ecde4450ff4e4b" exitCode=0
Nov 03 17:55:27.635044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:27.634584    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd" event={"ID":"f3201103-f5be-48f9-95fa-e61ed2602ad5","Type":"ContainerDied","Data":"ceae5a62a99192cdcdf2fae8ea86758b348dae59338ef31783ecde4450ff4e4b"}
Nov 03 17:55:28.115400 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.115368    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"]
Nov 03 17:55:28.119287 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.119262    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.179997 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.179953    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"]
Nov 03 17:55:28.225722 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.225690    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"distinct-2-w3-776ba7-kserve-self-signed-certs\""
Nov 03 17:55:28.270706 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.270671    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-dshm\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.270975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.270916    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-model-cache\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.271064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.271004    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-home\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.271064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.271040    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tls-certs\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.271179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.271105    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tmp-dir\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.271179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.271153    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2j9h\" (UniqueName: \"kubernetes.io/projected/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-kube-api-access-j2j9h\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372123 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372026    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-model-cache\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372123 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372070    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-home\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372123 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372103    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tls-certs\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372152    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tmp-dir\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372187    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2j9h\" (UniqueName: \"kubernetes.io/projected/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-kube-api-access-j2j9h\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372227    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-dshm\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372550 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372513    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-model-cache\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372550 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372535    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-home\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.372657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.372591    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tmp-dir\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.374883 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.374840    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-dshm\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.375041 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.375023    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tls-certs\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.442639 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.442608    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2j9h\" (UniqueName: \"kubernetes.io/projected/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-kube-api-access-j2j9h\") pod \"distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") " pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.732187 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.732149    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:28.781353 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.781320    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:28.878275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.878249    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-openshift-ca\") pod \"f3201103-f5be-48f9-95fa-e61ed2602ad5\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") "
Nov 03 17:55:28.878275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.878279    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-xks-ca\") pod \"f3201103-f5be-48f9-95fa-e61ed2602ad5\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") "
Nov 03 17:55:28.878511 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.878391    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3201103-f5be-48f9-95fa-e61ed2602ad5-kube-api-access-n4j9r\" (UniqueName: \"kubernetes.io/projected/f3201103-f5be-48f9-95fa-e61ed2602ad5-kube-api-access-n4j9r\") pod \"f3201103-f5be-48f9-95fa-e61ed2602ad5\" (UID: \"f3201103-f5be-48f9-95fa-e61ed2602ad5\") "
Nov 03 17:55:28.878930 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.878690    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-openshift-ca" pod "f3201103-f5be-48f9-95fa-e61ed2602ad5" (UID: "f3201103-f5be-48f9-95fa-e61ed2602ad5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:55:28.881192 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.881167    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-xks-ca" pod "f3201103-f5be-48f9-95fa-e61ed2602ad5" (UID: "f3201103-f5be-48f9-95fa-e61ed2602ad5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:55:28.881192 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.881185    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3201103-f5be-48f9-95fa-e61ed2602ad5-kube-api-access-n4j9r" pod "f3201103-f5be-48f9-95fa-e61ed2602ad5" (UID: "f3201103-f5be-48f9-95fa-e61ed2602ad5"). InnerVolumeSpecName "kube-api-access-n4j9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:55:28.979295 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.979267    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:55:28.979295 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.979295    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3201103-f5be-48f9-95fa-e61ed2602ad5-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:55:28.979486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:28.979309    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4j9r\" (UniqueName: \"kubernetes.io/projected/f3201103-f5be-48f9-95fa-e61ed2602ad5-kube-api-access-n4j9r\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:55:29.114014 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:55:29.113990    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe9f9fb3_acd8_486c_a33c_51c0c8a8f225.slice/crio-2be4292daab8898489e8d2ca369c63f2a0181357ebd19cd0b1f4ab61d44f48f0 WatchSource:0}: Error finding container 2be4292daab8898489e8d2ca369c63f2a0181357ebd19cd0b1f4ab61d44f48f0: Status 404 returned error can't find the container with id 2be4292daab8898489e8d2ca369c63f2a0181357ebd19cd0b1f4ab61d44f48f0
Nov 03 17:55:29.114229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.114205    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"]
Nov 03 17:55:29.648491 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.648391    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" event={"ID":"be9f9fb3-acd8-486c-a33c-51c0c8a8f225","Type":"ContainerStarted","Data":"2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67"}
Nov 03 17:55:29.648491 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.648435    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" event={"ID":"be9f9fb3-acd8-486c-a33c-51c0c8a8f225","Type":"ContainerStarted","Data":"2be4292daab8898489e8d2ca369c63f2a0181357ebd19cd0b1f4ab61d44f48f0"}
Nov 03 17:55:29.649748 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.649725    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd" event={"ID":"f3201103-f5be-48f9-95fa-e61ed2602ad5","Type":"ContainerDied","Data":"f0d65c6aa241d4ef1fd0fd3283099f0399c24cd726c6bd3c44097f1f459acb0a"}
Nov 03 17:55:29.649918 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.649763    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0d65c6aa241d4ef1fd0fd3283099f0399c24cd726c6bd3c44097f1f459acb0a"
Nov 03 17:55:29.649918 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.649763    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"
Nov 03 17:55:29.733045 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.733014    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:29.805273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:29.805222    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podStartSLOduration=1.805208836 podStartE2EDuration="1.805208836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:29.803600646 +0000 UTC m=+1309.262771644" watchObservedRunningTime="2026-11-03 17:55:29.805208836 +0000 UTC m=+1309.264379822"
Nov 03 17:55:36.992912 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:55:36.992852    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:55:38.732555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:38.732521    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:38.745022 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:38.744993    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:39.720493 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:39.720467    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:55:40.097734 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:40.097655    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"]
Nov 03 17:55:40.100846 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:40.100827    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d1f8ba9-2knq7"]
Nov 03 17:55:40.719413 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:40.719379    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58bb9a7-c060-4c3e-8ff7-99d4c9884e62" path="/var/lib/kubelet/pods/f58bb9a7-c060-4c3e-8ff7-99d4c9884e62/volumes"
Nov 03 17:55:49.060002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:49.059963    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"]
Nov 03 17:55:49.110724 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:49.110688    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc171688-7mcvj"]
Nov 03 17:55:50.712357 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:50.712321    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f19b936-6331-49d1-9eeb-38b3f85e4b63" path="/var/lib/kubelet/pods/7f19b936-6331-49d1-9eeb-38b3f85e4b63/volumes"
Nov 03 17:55:55.060553 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:55.060511    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"]
Nov 03 17:55:55.110965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:55.110933    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6090a2b-rv4qz"]
Nov 03 17:55:56.712911 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:55:56.712856    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6210aa8f-2dc6-49f8-b499-81e61b9fea02" path="/var/lib/kubelet/pods/6210aa8f-2dc6-49f8-b499-81e61b9fea02/volumes"
Nov 03 17:56:00.068307 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:00.068271    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"]
Nov 03 17:56:00.118966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:00.118933    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3447050f-662tr"]
Nov 03 17:56:00.713084 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:00.713039    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa2e595-aa5f-4e31-9298-9a269ba3b141" path="/var/lib/kubelet/pods/7fa2e595-aa5f-4e31-9298-9a269ba3b141/volumes"
Nov 03 17:56:05.230040 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.230006    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"]
Nov 03 17:56:05.230489 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.230475    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3201103-f5be-48f9-95fa-e61ed2602ad5" containerName="revoke-keys"
Nov 03 17:56:05.230533 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.230490    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3201103-f5be-48f9-95fa-e61ed2602ad5" containerName="revoke-keys"
Nov 03 17:56:05.230593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.230584    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3201103-f5be-48f9-95fa-e61ed2602ad5" containerName="revoke-keys"
Nov 03 17:56:05.233939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.233921    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.286327 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.286299    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jjn\" (UniqueName: \"kubernetes.io/projected/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-kube-api-access-c5jjn\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.286493 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.286335    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-home\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.286493 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.286368    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-dshm\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.286581 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.286543    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tls-certs\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.286710 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.286689    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tmp-dir\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.286780 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.286741    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-model-cache\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.338556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.338523    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"distinct-2-w5-548ca1-kserve-self-signed-certs\""
Nov 03 17:56:05.341787 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.341766    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"]
Nov 03 17:56:05.388457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.388425    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tmp-dir\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.388651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.388469    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-model-cache\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.388651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.388526    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jjn\" (UniqueName: \"kubernetes.io/projected/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-kube-api-access-c5jjn\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.388651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.388558    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-home\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.388651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.388604    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-dshm\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.388991 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.388653    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tls-certs\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.388991 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.388850    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tmp-dir\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.389101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.389050    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-home\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.389159 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.389122    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-model-cache\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.391296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.391268    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-dshm\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.391564 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.391545    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tls-certs\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.452862 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.452830    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jjn\" (UniqueName: \"kubernetes.io/projected/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-kube-api-access-c5jjn\") pod \"distinct-2-w5-548ca1-kserve-5f85557885-8d5h4\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") " pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.546111 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.546025    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:05.943972 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:05.943946    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"]
Nov 03 17:56:05.945434 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:05.945393    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod503a8b5b_5bf6_4f06_8268_3cda7c14b9ae.slice/crio-873cb354d6224402dc213e0a7bc96129641557dba7b8d3f9d69f75227d0b8ff9 WatchSource:0}: Error finding container 873cb354d6224402dc213e0a7bc96129641557dba7b8d3f9d69f75227d0b8ff9: Status 404 returned error can't find the container with id 873cb354d6224402dc213e0a7bc96129641557dba7b8d3f9d69f75227d0b8ff9
Nov 03 17:56:06.886929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:06.886894    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" event={"ID":"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae","Type":"ContainerStarted","Data":"828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e"}
Nov 03 17:56:06.887361 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:06.886936    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" event={"ID":"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae","Type":"ContainerStarted","Data":"873cb354d6224402dc213e0a7bc96129641557dba7b8d3f9d69f75227d0b8ff9"}
Nov 03 17:56:06.996426 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:56:06.993569    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:56:07.016299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:07.016252    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podStartSLOduration=2.016237052 podStartE2EDuration="2.016237052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:07.014004947 +0000 UTC m=+1346.473175933" watchObservedRunningTime="2026-11-03 17:56:07.016237052 +0000 UTC m=+1346.475408037"
Nov 03 17:56:07.546401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:07.546367    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:08.032229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.032191    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"]
Nov 03 17:56:08.036036 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.036021    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118377    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118429    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118518    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118545    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118599    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118650    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118739    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bsw\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-kube-api-access-p6bsw\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118779    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.123909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.118800    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220310 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220268    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220360    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bsw\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-kube-api-access-p6bsw\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220409    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220642 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220530    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220642 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220595    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220642 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220630    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220723    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220749    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220807    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.220921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220840    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.221036 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.220978    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.221036 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.221004    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.221188 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.221167    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.221453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.221430    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.223037 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.223019    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.223269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.223249    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.254254 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.254221    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"]
Nov 03 17:56:08.285021 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.284947    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.285021 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.285020    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bsw\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-kube-api-access-p6bsw\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.349363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.349329    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:08.787468 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.787443    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"]
Nov 03 17:56:08.787745 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:08.787722    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcac96ab_c8c5_4b4a_b5e6_6575bec12de2.slice/crio-bf67ae29a51493f1386ea1d73afae725f07e66cb741f51708e83dc3035dd3311 WatchSource:0}: Error finding container bf67ae29a51493f1386ea1d73afae725f07e66cb741f51708e83dc3035dd3311: Status 404 returned error can't find the container with id bf67ae29a51493f1386ea1d73afae725f07e66cb741f51708e83dc3035dd3311
Nov 03 17:56:08.790242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.790208    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:08.790331 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.790281    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:08.790331 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.790310    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:08.901259 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.901217    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" event={"ID":"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2","Type":"ContainerStarted","Data":"042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a"}
Nov 03 17:56:08.901451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:08.901287    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" event={"ID":"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2","Type":"ContainerStarted","Data":"bf67ae29a51493f1386ea1d73afae725f07e66cb741f51708e83dc3035dd3311"}
Nov 03 17:56:09.031126 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:09.031059    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podStartSLOduration=2.031038714 podStartE2EDuration="2.031038714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:09.026220346 +0000 UTC m=+1348.485391348" watchObservedRunningTime="2026-11-03 17:56:09.031038714 +0000 UTC m=+1348.490209701"
Nov 03 17:56:09.349553 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:09.349512    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:09.351338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:09.351314    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 17:56:09.351448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:09.351370    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 17:56:10.352403 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:10.352354    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 17:56:10.352952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:10.352436    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 17:56:11.350178 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:11.350148    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 17:56:11.350337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:11.350235    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 17:56:12.062612 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.062566    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"]
Nov 03 17:56:12.113824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.113791    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a97b68fc-zq87f"]
Nov 03 17:56:12.350629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.350509    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 17:56:12.350782 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.350630    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 17:56:12.714184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.714151    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f952732-7a64-4e0f-98d4-033d8c2c8fea" path="/var/lib/kubelet/pods/9f952732-7a64-4e0f-98d4-033d8c2c8fea/volumes"
Nov 03 17:56:12.775711 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.775671    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"]
Nov 03 17:56:12.779837 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.779817    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.852528 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.850003    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"]
Nov 03 17:56:12.887802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.887706    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7djl\" (UniqueName: \"kubernetes.io/projected/2a376f4a-0dda-4f61-9588-c9bf339184f2-kube-api-access-j7djl\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.888160 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.888136    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.888468 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.888406    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.890301 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.890043    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:56:12.890301 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.890161    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:56:12.890301 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.890223    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:56:12.890301 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.890269    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:56:12.989460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.989371    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7djl\" (UniqueName: \"kubernetes.io/projected/2a376f4a-0dda-4f61-9588-c9bf339184f2-kube-api-access-j7djl\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.989460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.989439    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.989710 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.989480    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.990182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.990157    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:12.992007 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:12.991988    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:13.058491 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:13.058456    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7djl\" (UniqueName: \"kubernetes.io/projected/2a376f4a-0dda-4f61-9588-c9bf339184f2-kube-api-access-j7djl\") pod \"maas-api-revoke-sub-c1dff07d-5wb6l\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:13.093429 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:13.093399    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:13.350270 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:13.350191    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 17:56:13.350270 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:13.350246    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 17:56:13.544169 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:13.544141    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"]
Nov 03 17:56:13.544407 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:13.544380    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a376f4a_0dda_4f61_9588_c9bf339184f2.slice/crio-2d9514a8cf730095c778bc2f1373f923d4f315071669dc882725b40ef2c2dbc5 WatchSource:0}: Error finding container 2d9514a8cf730095c778bc2f1373f923d4f315071669dc882725b40ef2c2dbc5: Status 404 returned error can't find the container with id 2d9514a8cf730095c778bc2f1373f923d4f315071669dc882725b40ef2c2dbc5
Nov 03 17:56:13.953076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:13.953042    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l" event={"ID":"2a376f4a-0dda-4f61-9588-c9bf339184f2","Type":"ContainerStarted","Data":"5b57d50d1efa3062374e57d2066736eda28be43aac0be9d69080545d5d4d5e80"}
Nov 03 17:56:13.953279 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:13.953095    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l" event={"ID":"2a376f4a-0dda-4f61-9588-c9bf339184f2","Type":"ContainerStarted","Data":"2d9514a8cf730095c778bc2f1373f923d4f315071669dc882725b40ef2c2dbc5"}
Nov 03 17:56:14.075041 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:14.074943    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l" podStartSLOduration=2.074918367 podStartE2EDuration="2.074918367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:14.072794547 +0000 UTC m=+1353.531965542" watchObservedRunningTime="2026-11-03 17:56:14.074918367 +0000 UTC m=+1353.534089353"
Nov 03 17:56:14.350701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:14.350609    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 17:56:14.350701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:14.350666    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 17:56:14.962355 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:14.962267    4493 generic.go:356] "Generic (PLEG): container finished" podID="2a376f4a-0dda-4f61-9588-c9bf339184f2" containerID="5b57d50d1efa3062374e57d2066736eda28be43aac0be9d69080545d5d4d5e80" exitCode=0
Nov 03 17:56:14.962505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:14.962366    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l" event={"ID":"2a376f4a-0dda-4f61-9588-c9bf339184f2","Type":"ContainerDied","Data":"5b57d50d1efa3062374e57d2066736eda28be43aac0be9d69080545d5d4d5e80"}
Nov 03 17:56:15.349745 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:15.349708    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 17:56:15.349961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:15.349768    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 17:56:15.547312 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:15.547270    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:15.561366 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:15.561340    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:15.988164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:15.988132    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:56:16.137498 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.137467    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:16.232424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.232343    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-xks-ca\") pod \"2a376f4a-0dda-4f61-9588-c9bf339184f2\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") "
Nov 03 17:56:16.232590 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.232444    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a376f4a-0dda-4f61-9588-c9bf339184f2-kube-api-access-j7djl\" (UniqueName: \"kubernetes.io/projected/2a376f4a-0dda-4f61-9588-c9bf339184f2-kube-api-access-j7djl\") pod \"2a376f4a-0dda-4f61-9588-c9bf339184f2\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") "
Nov 03 17:56:16.232590 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.232571    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-openshift-ca\") pod \"2a376f4a-0dda-4f61-9588-c9bf339184f2\" (UID: \"2a376f4a-0dda-4f61-9588-c9bf339184f2\") "
Nov 03 17:56:16.232975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.232946    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-openshift-ca" pod "2a376f4a-0dda-4f61-9588-c9bf339184f2" (UID: "2a376f4a-0dda-4f61-9588-c9bf339184f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:16.234761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.234739    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a376f4a-0dda-4f61-9588-c9bf339184f2-kube-api-access-j7djl" pod "2a376f4a-0dda-4f61-9588-c9bf339184f2" (UID: "2a376f4a-0dda-4f61-9588-c9bf339184f2"). InnerVolumeSpecName "kube-api-access-j7djl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:16.234898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.234855    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-xks-ca" pod "2a376f4a-0dda-4f61-9588-c9bf339184f2" (UID: "2a376f4a-0dda-4f61-9588-c9bf339184f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:56:16.334130 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.334089    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7djl\" (UniqueName: \"kubernetes.io/projected/2a376f4a-0dda-4f61-9588-c9bf339184f2-kube-api-access-j7djl\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:16.334130 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.334128    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:16.334130 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.334138    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a376f4a-0dda-4f61-9588-c9bf339184f2-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:16.354859 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.354828    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:16.355209 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.355190    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:16.356008 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.355990    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 17:56:16.978465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.978377    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"
Nov 03 17:56:16.978465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.978421    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l" event={"ID":"2a376f4a-0dda-4f61-9588-c9bf339184f2","Type":"ContainerDied","Data":"2d9514a8cf730095c778bc2f1373f923d4f315071669dc882725b40ef2c2dbc5"}
Nov 03 17:56:16.978995 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:16.978497    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d9514a8cf730095c778bc2f1373f923d4f315071669dc882725b40ef2c2dbc5"
Nov 03 17:56:21.882665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.882631    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"]
Nov 03 17:56:21.883143 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.883128    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2a376f4a-0dda-4f61-9588-c9bf339184f2" containerName="revoke-keys"
Nov 03 17:56:21.883143 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.883143    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a376f4a-0dda-4f61-9588-c9bf339184f2" containerName="revoke-keys"
Nov 03 17:56:21.883249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.883239    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a376f4a-0dda-4f61-9588-c9bf339184f2" containerName="revoke-keys"
Nov 03 17:56:21.886684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.886664    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:21.993068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.993038    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:56:21.993068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.993040    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:56:21.993296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.993074    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:56:21.993296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.993107    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:56:21.996275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.996252    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"]
Nov 03 17:56:21.997107 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.997091    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:21.997282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.997265    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:21.997324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:21.997298    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdgk\" (UniqueName: \"kubernetes.io/projected/bf615a6f-c67c-4eef-98bd-f411b7f37fac-kube-api-access-jxdgk\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.098906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.098852    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.099083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.098931    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdgk\" (UniqueName: \"kubernetes.io/projected/bf615a6f-c67c-4eef-98bd-f411b7f37fac-kube-api-access-jxdgk\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.099083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.099039    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.099730 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.099646    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.101470 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.101446    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.162984 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.162917    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdgk\" (UniqueName: \"kubernetes.io/projected/bf615a6f-c67c-4eef-98bd-f411b7f37fac-kube-api-access-jxdgk\") pod \"maas-api-revoke-sub-6e14f778-dllmd\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.199181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.199149    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:22.579138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:22.579106    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"]
Nov 03 17:56:22.579923 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:22.579897    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf615a6f_c67c_4eef_98bd_f411b7f37fac.slice/crio-1da65c85940f4c31b989aa6980bae495da09bf4fae6ca13b539d8133afff3bc9 WatchSource:0}: Error finding container 1da65c85940f4c31b989aa6980bae495da09bf4fae6ca13b539d8133afff3bc9: Status 404 returned error can't find the container with id 1da65c85940f4c31b989aa6980bae495da09bf4fae6ca13b539d8133afff3bc9
Nov 03 17:56:23.017658 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.017619    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd" event={"ID":"bf615a6f-c67c-4eef-98bd-f411b7f37fac","Type":"ContainerStarted","Data":"f77cd1a9be8317ec0e9ee3f82fb2e50403bb06bc8fe333970804fafdb4841b4e"}
Nov 03 17:56:23.018179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.017679    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd" event={"ID":"bf615a6f-c67c-4eef-98bd-f411b7f37fac","Type":"ContainerStarted","Data":"1da65c85940f4c31b989aa6980bae495da09bf4fae6ca13b539d8133afff3bc9"}
Nov 03 17:56:23.085948 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.085860    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"]
Nov 03 17:56:23.093272 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.093237    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"]
Nov 03 17:56:23.096062 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.096035    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dba8f0-htz6v"]
Nov 03 17:56:23.109395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.109359    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f82e893-l59lk"]
Nov 03 17:56:23.160165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.160113    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd" podStartSLOduration=2.160100875 podStartE2EDuration="2.160100875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:23.156282951 +0000 UTC m=+1362.615453937" watchObservedRunningTime="2026-11-03 17:56:23.160100875 +0000 UTC m=+1362.619271906"
Nov 03 17:56:23.982818 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.982783    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"]
Nov 03 17:56:23.987219 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:23.987195    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.025565 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.025533    4493 generic.go:356] "Generic (PLEG): container finished" podID="bf615a6f-c67c-4eef-98bd-f411b7f37fac" containerID="f77cd1a9be8317ec0e9ee3f82fb2e50403bb06bc8fe333970804fafdb4841b4e" exitCode=0
Nov 03 17:56:24.025941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.025623    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd" event={"ID":"bf615a6f-c67c-4eef-98bd-f411b7f37fac","Type":"ContainerDied","Data":"f77cd1a9be8317ec0e9ee3f82fb2e50403bb06bc8fe333970804fafdb4841b4e"}
Nov 03 17:56:24.074798 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.074762    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"]
Nov 03 17:56:24.094266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.094234    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-42c162-openshift-default-dockercfg-jf8vf\""
Nov 03 17:56:24.122711 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.122678    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.122904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.122732    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.122904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.122759    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.122904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.122897    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.123096 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.122921    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.123096 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.122982    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.123096 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.123034    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.123242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.123167    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp7rr\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-kube-api-access-xp7rr\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.123242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.123196    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.224775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.224737    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.224775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.224778    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225011 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.224838    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225011 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.224899    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225011 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.224959    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp7rr\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-kube-api-access-xp7rr\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225011 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.224986    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225067    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225121    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225159    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225441    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225582 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225557    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225709 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225580    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225709 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225597    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.225787 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.225716    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.227498 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.227477    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.227619 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.227602    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.289221 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.289153    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.289221 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.289154    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp7rr\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-kube-api-access-xp7rr\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.302119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.302088    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:24.715938 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.713496    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cdfe4c-f754-40bf-a994-30321ea9b764" path="/var/lib/kubelet/pods/d2cdfe4c-f754-40bf-a994-30321ea9b764/volumes"
Nov 03 17:56:24.715938 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.714113    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b42601-ec89-4d26-8f21-d4cdb0e4623e" path="/var/lib/kubelet/pods/d4b42601-ec89-4d26-8f21-d4cdb0e4623e/volumes"
Nov 03 17:56:24.747143 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.747114    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"]
Nov 03 17:56:24.748141 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:24.748116    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40c3dc2_b5ce_4570_b0ab_4d441aa89e8c.slice/crio-dc0c434b0a4b9f876cbba6261ab496021571717130e1cfa0526da552793e2251 WatchSource:0}: Error finding container dc0c434b0a4b9f876cbba6261ab496021571717130e1cfa0526da552793e2251: Status 404 returned error can't find the container with id dc0c434b0a4b9f876cbba6261ab496021571717130e1cfa0526da552793e2251
Nov 03 17:56:24.750537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.750508    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:24.750610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.750575    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:24.750610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:24.750603    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:25.034006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.033903    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" event={"ID":"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c","Type":"ContainerStarted","Data":"75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6"}
Nov 03 17:56:25.034006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.033976    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" event={"ID":"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c","Type":"ContainerStarted","Data":"dc0c434b0a4b9f876cbba6261ab496021571717130e1cfa0526da552793e2251"}
Nov 03 17:56:25.161953 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.161896    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podStartSLOduration=2.161861631 podStartE2EDuration="2.161861631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:25.159200077 +0000 UTC m=+1364.618371062" watchObservedRunningTime="2026-11-03 17:56:25.161861631 +0000 UTC m=+1364.621032673"
Nov 03 17:56:25.162556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.162534    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:25.237226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.237164    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-xks-ca\") pod \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") "
Nov 03 17:56:25.237226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.237214    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-openshift-ca\") pod \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") "
Nov 03 17:56:25.237396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.237381    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf615a6f-c67c-4eef-98bd-f411b7f37fac-kube-api-access-jxdgk\" (UniqueName: \"kubernetes.io/projected/bf615a6f-c67c-4eef-98bd-f411b7f37fac-kube-api-access-jxdgk\") pod \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\" (UID: \"bf615a6f-c67c-4eef-98bd-f411b7f37fac\") "
Nov 03 17:56:25.237642 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.237615    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-openshift-ca" pod "bf615a6f-c67c-4eef-98bd-f411b7f37fac" (UID: "bf615a6f-c67c-4eef-98bd-f411b7f37fac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:25.238064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.238040    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:25.241471 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.241422    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-xks-ca" pod "bf615a6f-c67c-4eef-98bd-f411b7f37fac" (UID: "bf615a6f-c67c-4eef-98bd-f411b7f37fac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:56:25.241579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.241493    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf615a6f-c67c-4eef-98bd-f411b7f37fac-kube-api-access-jxdgk" pod "bf615a6f-c67c-4eef-98bd-f411b7f37fac" (UID: "bf615a6f-c67c-4eef-98bd-f411b7f37fac"). InnerVolumeSpecName "kube-api-access-jxdgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:25.302714 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.302636    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:25.304049 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.304019    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:25.304160 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.304076    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 17:56:25.339335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.339311    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf615a6f-c67c-4eef-98bd-f411b7f37fac-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:25.339335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:25.339336    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdgk\" (UniqueName: \"kubernetes.io/projected/bf615a6f-c67c-4eef-98bd-f411b7f37fac-kube-api-access-jxdgk\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:26.048183 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:26.048151    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"
Nov 03 17:56:26.048669 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:26.048183    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd" event={"ID":"bf615a6f-c67c-4eef-98bd-f411b7f37fac","Type":"ContainerDied","Data":"1da65c85940f4c31b989aa6980bae495da09bf4fae6ca13b539d8133afff3bc9"}
Nov 03 17:56:26.048669 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:26.048231    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1da65c85940f4c31b989aa6980bae495da09bf4fae6ca13b539d8133afff3bc9"
Nov 03 17:56:26.303135 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:26.303053    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:26.303135 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:26.303112    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 17:56:27.302935 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:27.302900    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:27.303386 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:27.302964    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 17:56:28.302974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:28.302936    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:28.303365 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:28.302995    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 17:56:29.303262 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:29.303223    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:29.303646 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:29.303285    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 17:56:30.303243 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:30.303210    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:30.303472 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:30.303260    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 17:56:31.303330 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:31.303290    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 17:56:31.303637 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:31.303366    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 17:56:32.062675 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:32.062625    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"]
Nov 03 17:56:32.113234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:32.113200    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-789eaf37-gxf6r"]
Nov 03 17:56:32.472216 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:32.472181    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:32.472742 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:32.472721    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:32.473599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:32.473580    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 17:56:32.712219 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:32.712182    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16057974-779c-4803-b839-7a41487be851" path="/var/lib/kubelet/pods/16057974-779c-4803-b839-7a41487be851/volumes"
Nov 03 17:56:35.785052 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.785018    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"]
Nov 03 17:56:35.785521 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.785507    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bf615a6f-c67c-4eef-98bd-f411b7f37fac" containerName="revoke-keys"
Nov 03 17:56:35.785587 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.785523    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf615a6f-c67c-4eef-98bd-f411b7f37fac" containerName="revoke-keys"
Nov 03 17:56:35.785638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.785628    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf615a6f-c67c-4eef-98bd-f411b7f37fac" containerName="revoke-keys"
Nov 03 17:56:35.789273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.789246    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.849085 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.849054    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"]
Nov 03 17:56:35.864307 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.864267    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxsxv\" (UniqueName: \"kubernetes.io/projected/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-kube-api-access-fxsxv\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.864509 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.864352    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.864593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.864569    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.895248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.895208    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:56:35.895430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.895412    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:56:35.895430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.895412    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:56:35.895554 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.895431    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:56:35.965740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.965711    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.965933 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.965825    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxsxv\" (UniqueName: \"kubernetes.io/projected/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-kube-api-access-fxsxv\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.966030 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.966004    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.966508 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.966489    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:35.968494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:35.968470    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:36.030416 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:36.030384    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxsxv\" (UniqueName: \"kubernetes.io/projected/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-kube-api-access-fxsxv\") pod \"maas-api-revoke-sub-a654930f-cdfft\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:36.103596 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:36.103524    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:36.497281 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:36.497251    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"]
Nov 03 17:56:36.498919 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:36.498860    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c0579e_477e_4a21_9e1b_ac2bd06f6164.slice/crio-f0f1ca21accf6e07f06ca04ff9f6f5db02b20a2a8db183dea8dbab0c77071b84 WatchSource:0}: Error finding container f0f1ca21accf6e07f06ca04ff9f6f5db02b20a2a8db183dea8dbab0c77071b84: Status 404 returned error can't find the container with id f0f1ca21accf6e07f06ca04ff9f6f5db02b20a2a8db183dea8dbab0c77071b84
Nov 03 17:56:36.997321 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:56:36.997290    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:56:37.135909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.135808    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft" event={"ID":"c8c0579e-477e-4a21-9e1b-ac2bd06f6164","Type":"ContainerStarted","Data":"24c154963a9996f84fdcb23e3d7c0c1f46bc75b955541c20961fdcdb2c1eda57"}
Nov 03 17:56:37.135909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.135904    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft" event={"ID":"c8c0579e-477e-4a21-9e1b-ac2bd06f6164","Type":"ContainerStarted","Data":"f0f1ca21accf6e07f06ca04ff9f6f5db02b20a2a8db183dea8dbab0c77071b84"}
Nov 03 17:56:37.257616 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.257566    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft" podStartSLOduration=2.257549008 podStartE2EDuration="2.257549008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:37.257427127 +0000 UTC m=+1376.716598113" watchObservedRunningTime="2026-11-03 17:56:37.257549008 +0000 UTC m=+1376.716719993"
Nov 03 17:56:37.618310 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.618277    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"]
Nov 03 17:56:37.622154 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.622138    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.682538 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.682506    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"]
Nov 03 17:56:37.789012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.788922    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.789157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.789035    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.789224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.789152    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffbm\" (UniqueName: \"kubernetes.io/projected/2ff29423-9286-45fe-8b42-451d6117cdd8-kube-api-access-gffbm\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.890923 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.890890    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.891125 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.890963    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.891125 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.891036    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffbm\" (UniqueName: \"kubernetes.io/projected/2ff29423-9286-45fe-8b42-451d6117cdd8-kube-api-access-gffbm\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.891594 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.891566    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.893451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.893433    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:37.955152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:37.955124    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffbm\" (UniqueName: \"kubernetes.io/projected/2ff29423-9286-45fe-8b42-451d6117cdd8-kube-api-access-gffbm\") pod \"maas-api-revoke-sub-feae5603-dkvl9\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:38.143802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.143717    4493 generic.go:356] "Generic (PLEG): container finished" podID="c8c0579e-477e-4a21-9e1b-ac2bd06f6164" containerID="24c154963a9996f84fdcb23e3d7c0c1f46bc75b955541c20961fdcdb2c1eda57" exitCode=0
Nov 03 17:56:38.144210 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.143802    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft" event={"ID":"c8c0579e-477e-4a21-9e1b-ac2bd06f6164","Type":"ContainerDied","Data":"24c154963a9996f84fdcb23e3d7c0c1f46bc75b955541c20961fdcdb2c1eda57"}
Nov 03 17:56:38.236261 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.236229    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:38.324243 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.324211    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"]
Nov 03 17:56:38.329646 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.329620    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.384382 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.384349    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"]
Nov 03 17:56:38.499539 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.499511    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swq78\" (UniqueName: \"kubernetes.io/projected/385e15fc-41a0-4ccc-8129-82036ab6f8eb-kube-api-access-swq78\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.499703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.499545    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.499703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.499640    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.600961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.600923    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swq78\" (UniqueName: \"kubernetes.io/projected/385e15fc-41a0-4ccc-8129-82036ab6f8eb-kube-api-access-swq78\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.600961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.600965    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.601157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.601032    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.601660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.601639    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.603512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.603494    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.621633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.621606    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"]
Nov 03 17:56:38.622299 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:38.622279    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff29423_9286_45fe_8b42_451d6117cdd8.slice/crio-eb0e1f06ac51d35dd0b91e21a48568280136680328435ff166bd5db2cb14beeb WatchSource:0}: Error finding container eb0e1f06ac51d35dd0b91e21a48568280136680328435ff166bd5db2cb14beeb: Status 404 returned error can't find the container with id eb0e1f06ac51d35dd0b91e21a48568280136680328435ff166bd5db2cb14beeb
Nov 03 17:56:38.665606 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.665580    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swq78\" (UniqueName: \"kubernetes.io/projected/385e15fc-41a0-4ccc-8129-82036ab6f8eb-kube-api-access-swq78\") pod \"maas-api-revoke-sub-e64633c7-67twq\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:38.944553 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:38.944532    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:39.152110 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.152027    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9" event={"ID":"2ff29423-9286-45fe-8b42-451d6117cdd8","Type":"ContainerStarted","Data":"d8b1c73c8e180b1f5502d187187cf1418c445b69beac8f95f4169da224d32c7f"}
Nov 03 17:56:39.152490 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.152123    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9" event={"ID":"2ff29423-9286-45fe-8b42-451d6117cdd8","Type":"ContainerStarted","Data":"eb0e1f06ac51d35dd0b91e21a48568280136680328435ff166bd5db2cb14beeb"}
Nov 03 17:56:39.275449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.275388    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9" podStartSLOduration=2.2753690300000002 podStartE2EDuration="2.27536903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:39.274269237 +0000 UTC m=+1378.733440224" watchObservedRunningTime="2026-11-03 17:56:39.27536903 +0000 UTC m=+1378.734540018"
Nov 03 17:56:39.330005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.329981    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"]
Nov 03 17:56:39.332071 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:39.332042    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385e15fc_41a0_4ccc_8129_82036ab6f8eb.slice/crio-26a388c38a20065fb461d3c986af428e7c447cd9f48e5af0a1efedd70e1bc7fd WatchSource:0}: Error finding container 26a388c38a20065fb461d3c986af428e7c447cd9f48e5af0a1efedd70e1bc7fd: Status 404 returned error can't find the container with id 26a388c38a20065fb461d3c986af428e7c447cd9f48e5af0a1efedd70e1bc7fd
Nov 03 17:56:39.332190 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.332095    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:39.513629 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.513600    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-openshift-ca\") pod \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") "
Nov 03 17:56:39.513791 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.513737    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-kube-api-access-fxsxv\" (UniqueName: \"kubernetes.io/projected/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-kube-api-access-fxsxv\") pod \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") "
Nov 03 17:56:39.513791 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.513757    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-xks-ca\") pod \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\" (UID: \"c8c0579e-477e-4a21-9e1b-ac2bd06f6164\") "
Nov 03 17:56:39.514046 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.514022    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-openshift-ca" pod "c8c0579e-477e-4a21-9e1b-ac2bd06f6164" (UID: "c8c0579e-477e-4a21-9e1b-ac2bd06f6164"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:39.516052 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.515864    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-xks-ca" pod "c8c0579e-477e-4a21-9e1b-ac2bd06f6164" (UID: "c8c0579e-477e-4a21-9e1b-ac2bd06f6164"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:56:39.516052 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.516036    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-kube-api-access-fxsxv" pod "c8c0579e-477e-4a21-9e1b-ac2bd06f6164" (UID: "c8c0579e-477e-4a21-9e1b-ac2bd06f6164"). InnerVolumeSpecName "kube-api-access-fxsxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:39.615897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.615844    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxsxv\" (UniqueName: \"kubernetes.io/projected/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-kube-api-access-fxsxv\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:39.615991 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.615918    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:39.615991 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:39.615936    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c0579e-477e-4a21-9e1b-ac2bd06f6164-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:40.061515 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.061468    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"]
Nov 03 17:56:40.066755 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.066727    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c9bf09d-fqttg"]
Nov 03 17:56:40.160287 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.160249    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq" event={"ID":"385e15fc-41a0-4ccc-8129-82036ab6f8eb","Type":"ContainerStarted","Data":"eda052f40fd1c6c4292d8875eb90be8802d8da31d7ccbdf88568acd4a9a11613"}
Nov 03 17:56:40.160711 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.160308    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq" event={"ID":"385e15fc-41a0-4ccc-8129-82036ab6f8eb","Type":"ContainerStarted","Data":"26a388c38a20065fb461d3c986af428e7c447cd9f48e5af0a1efedd70e1bc7fd"}
Nov 03 17:56:40.161912 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.161884    4493 generic.go:356] "Generic (PLEG): container finished" podID="2ff29423-9286-45fe-8b42-451d6117cdd8" containerID="d8b1c73c8e180b1f5502d187187cf1418c445b69beac8f95f4169da224d32c7f" exitCode=0
Nov 03 17:56:40.162033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.161965    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9" event={"ID":"2ff29423-9286-45fe-8b42-451d6117cdd8","Type":"ContainerDied","Data":"d8b1c73c8e180b1f5502d187187cf1418c445b69beac8f95f4169da224d32c7f"}
Nov 03 17:56:40.163270 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.163252    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"
Nov 03 17:56:40.163352 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.163277    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft" event={"ID":"c8c0579e-477e-4a21-9e1b-ac2bd06f6164","Type":"ContainerDied","Data":"f0f1ca21accf6e07f06ca04ff9f6f5db02b20a2a8db183dea8dbab0c77071b84"}
Nov 03 17:56:40.163352 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.163306    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f1ca21accf6e07f06ca04ff9f6f5db02b20a2a8db183dea8dbab0c77071b84"
Nov 03 17:56:40.282414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.282363    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq" podStartSLOduration=2.282349552 podStartE2EDuration="2.282349552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:40.280523299 +0000 UTC m=+1379.739694347" watchObservedRunningTime="2026-11-03 17:56:40.282349552 +0000 UTC m=+1379.741520538"
Nov 03 17:56:40.713635 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.713599    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056435d9-5dee-4e5c-8508-ae9cfc456485" path="/var/lib/kubelet/pods/056435d9-5dee-4e5c-8508-ae9cfc456485/volumes"
Nov 03 17:56:40.884034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.884008    4493 scope.go:122] "RemoveContainer" containerID="3e46468cd59d9f410974b1a3d1490a1ae50dd61aaa2af3bd63851ed8921f540d"
Nov 03 17:56:40.893460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:40.893437    4493 scope.go:122] "RemoveContainer" containerID="f1ba214462c51b2e6ce2e80dc3a3f33bfbc673d4345aff71da52f12233097e90"
Nov 03 17:56:41.073701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.073613    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"]
Nov 03 17:56:41.124502 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.124468    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a150bb1b-4ztfc"]
Nov 03 17:56:41.170602 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.170573    4493 generic.go:356] "Generic (PLEG): container finished" podID="385e15fc-41a0-4ccc-8129-82036ab6f8eb" containerID="eda052f40fd1c6c4292d8875eb90be8802d8da31d7ccbdf88568acd4a9a11613" exitCode=0
Nov 03 17:56:41.171068 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.170680    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq" event={"ID":"385e15fc-41a0-4ccc-8129-82036ab6f8eb","Type":"ContainerDied","Data":"eda052f40fd1c6c4292d8875eb90be8802d8da31d7ccbdf88568acd4a9a11613"}
Nov 03 17:56:41.269465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.269429    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"]
Nov 03 17:56:41.270229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.270206    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c8c0579e-477e-4a21-9e1b-ac2bd06f6164" containerName="revoke-keys"
Nov 03 17:56:41.270229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.270228    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c0579e-477e-4a21-9e1b-ac2bd06f6164" containerName="revoke-keys"
Nov 03 17:56:41.270427 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.270390    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c0579e-477e-4a21-9e1b-ac2bd06f6164" containerName="revoke-keys"
Nov 03 17:56:41.276117 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.276067    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.323572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.323540    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:41.332474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.332201    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"]
Nov 03 17:56:41.439834 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.439799    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-xks-ca\") pod \"2ff29423-9286-45fe-8b42-451d6117cdd8\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") "
Nov 03 17:56:41.439834 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.439836    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-openshift-ca\") pod \"2ff29423-9286-45fe-8b42-451d6117cdd8\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") "
Nov 03 17:56:41.440095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.439930    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff29423-9286-45fe-8b42-451d6117cdd8-kube-api-access-gffbm\" (UniqueName: \"kubernetes.io/projected/2ff29423-9286-45fe-8b42-451d6117cdd8-kube-api-access-gffbm\") pod \"2ff29423-9286-45fe-8b42-451d6117cdd8\" (UID: \"2ff29423-9286-45fe-8b42-451d6117cdd8\") "
Nov 03 17:56:41.440266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.440239    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-openshift-ca" pod "2ff29423-9286-45fe-8b42-451d6117cdd8" (UID: "2ff29423-9286-45fe-8b42-451d6117cdd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:41.440266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.440251    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.440381 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.440363    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.440512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.440499    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swlx\" (UniqueName: \"kubernetes.io/projected/8a3dcaed-470a-4a37-8193-95005dae91e1-kube-api-access-9swlx\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.440638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.440624    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:41.442283 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.442233    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff29423-9286-45fe-8b42-451d6117cdd8-kube-api-access-gffbm" pod "2ff29423-9286-45fe-8b42-451d6117cdd8" (UID: "2ff29423-9286-45fe-8b42-451d6117cdd8"). InnerVolumeSpecName "kube-api-access-gffbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:41.442395 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.442325    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-xks-ca" pod "2ff29423-9286-45fe-8b42-451d6117cdd8" (UID: "2ff29423-9286-45fe-8b42-451d6117cdd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:56:41.541840 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.541806    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.542069 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.541977    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swlx\" (UniqueName: \"kubernetes.io/projected/8a3dcaed-470a-4a37-8193-95005dae91e1-kube-api-access-9swlx\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.542138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.542119    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.542236 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.542219    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff29423-9286-45fe-8b42-451d6117cdd8-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:41.542286 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.542242    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gffbm\" (UniqueName: \"kubernetes.io/projected/2ff29423-9286-45fe-8b42-451d6117cdd8-kube-api-access-gffbm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:41.542572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.542554    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.544723 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.544696    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.605936 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.605854    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swlx\" (UniqueName: \"kubernetes.io/projected/8a3dcaed-470a-4a37-8193-95005dae91e1-kube-api-access-9swlx\") pod \"maas-api-revoke-sub-f8551fa4-8548d\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:41.890570 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:41.890483    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:42.178208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.178130    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"
Nov 03 17:56:42.178208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.178173    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9" event={"ID":"2ff29423-9286-45fe-8b42-451d6117cdd8","Type":"ContainerDied","Data":"eb0e1f06ac51d35dd0b91e21a48568280136680328435ff166bd5db2cb14beeb"}
Nov 03 17:56:42.178621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.178229    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb0e1f06ac51d35dd0b91e21a48568280136680328435ff166bd5db2cb14beeb"
Nov 03 17:56:42.275525 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:42.275484    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3dcaed_470a_4a37_8193_95005dae91e1.slice/crio-cdf59d22f8864a21c8fd39e352ca8722e2726b90a789266a9d0fbdb81190e52e WatchSource:0}: Error finding container cdf59d22f8864a21c8fd39e352ca8722e2726b90a789266a9d0fbdb81190e52e: Status 404 returned error can't find the container with id cdf59d22f8864a21c8fd39e352ca8722e2726b90a789266a9d0fbdb81190e52e
Nov 03 17:56:42.295698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.295671    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"]
Nov 03 17:56:42.344678 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.344654    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:42.453000 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.452913    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-xks-ca\") pod \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") "
Nov 03 17:56:42.453148 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.453009    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-openshift-ca\") pod \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") "
Nov 03 17:56:42.453148 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.453084    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385e15fc-41a0-4ccc-8129-82036ab6f8eb-kube-api-access-swq78\" (UniqueName: \"kubernetes.io/projected/385e15fc-41a0-4ccc-8129-82036ab6f8eb-kube-api-access-swq78\") pod \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\" (UID: \"385e15fc-41a0-4ccc-8129-82036ab6f8eb\") "
Nov 03 17:56:42.453456 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.453425    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-openshift-ca" pod "385e15fc-41a0-4ccc-8129-82036ab6f8eb" (UID: "385e15fc-41a0-4ccc-8129-82036ab6f8eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:42.455269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.455241    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-xks-ca" pod "385e15fc-41a0-4ccc-8129-82036ab6f8eb" (UID: "385e15fc-41a0-4ccc-8129-82036ab6f8eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:56:42.455375 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.455314    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385e15fc-41a0-4ccc-8129-82036ab6f8eb-kube-api-access-swq78" pod "385e15fc-41a0-4ccc-8129-82036ab6f8eb" (UID: "385e15fc-41a0-4ccc-8129-82036ab6f8eb"). InnerVolumeSpecName "kube-api-access-swq78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:42.554389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.554351    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swq78\" (UniqueName: \"kubernetes.io/projected/385e15fc-41a0-4ccc-8129-82036ab6f8eb-kube-api-access-swq78\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:42.554389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.554382    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:42.554389 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.554391    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/385e15fc-41a0-4ccc-8129-82036ab6f8eb-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:42.713394 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:42.713352    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ea6215-0ad8-4229-8c51-f6c1db7861d2" path="/var/lib/kubelet/pods/49ea6215-0ad8-4229-8c51-f6c1db7861d2/volumes"
Nov 03 17:56:43.187363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:43.187342    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"
Nov 03 17:56:43.187693 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:43.187351    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq" event={"ID":"385e15fc-41a0-4ccc-8129-82036ab6f8eb","Type":"ContainerDied","Data":"26a388c38a20065fb461d3c986af428e7c447cd9f48e5af0a1efedd70e1bc7fd"}
Nov 03 17:56:43.187693 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:43.187410    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a388c38a20065fb461d3c986af428e7c447cd9f48e5af0a1efedd70e1bc7fd"
Nov 03 17:56:43.188738 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:43.188715    4493 generic.go:356] "Generic (PLEG): container finished" podID="8a3dcaed-470a-4a37-8193-95005dae91e1" containerID="8523777bf209bf4395039bb4d396c6745239b35246d53d13b91dd983fd10c715" exitCode=0
Nov 03 17:56:43.188823 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:43.188754    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d" event={"ID":"8a3dcaed-470a-4a37-8193-95005dae91e1","Type":"ContainerDied","Data":"8523777bf209bf4395039bb4d396c6745239b35246d53d13b91dd983fd10c715"}
Nov 03 17:56:43.188823 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:43.188800    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d" event={"ID":"8a3dcaed-470a-4a37-8193-95005dae91e1","Type":"ContainerStarted","Data":"cdf59d22f8864a21c8fd39e352ca8722e2726b90a789266a9d0fbdb81190e52e"}
Nov 03 17:56:44.347613 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.347587    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:44.372055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.372028    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-xks-ca\") pod \"8a3dcaed-470a-4a37-8193-95005dae91e1\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") "
Nov 03 17:56:44.372189 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.372133    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3dcaed-470a-4a37-8193-95005dae91e1-kube-api-access-9swlx\" (UniqueName: \"kubernetes.io/projected/8a3dcaed-470a-4a37-8193-95005dae91e1-kube-api-access-9swlx\") pod \"8a3dcaed-470a-4a37-8193-95005dae91e1\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") "
Nov 03 17:56:44.372293 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.372275    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-openshift-ca\") pod \"8a3dcaed-470a-4a37-8193-95005dae91e1\" (UID: \"8a3dcaed-470a-4a37-8193-95005dae91e1\") "
Nov 03 17:56:44.372663 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.372629    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-openshift-ca" pod "8a3dcaed-470a-4a37-8193-95005dae91e1" (UID: "8a3dcaed-470a-4a37-8193-95005dae91e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:44.374424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.374404    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3dcaed-470a-4a37-8193-95005dae91e1-kube-api-access-9swlx" pod "8a3dcaed-470a-4a37-8193-95005dae91e1" (UID: "8a3dcaed-470a-4a37-8193-95005dae91e1"). InnerVolumeSpecName "kube-api-access-9swlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:44.374509 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.374483    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-xks-ca" pod "8a3dcaed-470a-4a37-8193-95005dae91e1" (UID: "8a3dcaed-470a-4a37-8193-95005dae91e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:56:44.474138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.474107    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swlx\" (UniqueName: \"kubernetes.io/projected/8a3dcaed-470a-4a37-8193-95005dae91e1-kube-api-access-9swlx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.474264 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.474142    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:44.474264 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:44.474156    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3dcaed-470a-4a37-8193-95005dae91e1-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:45.205170 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:45.205089    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"
Nov 03 17:56:45.205170 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:45.205118    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d" event={"ID":"8a3dcaed-470a-4a37-8193-95005dae91e1","Type":"ContainerDied","Data":"cdf59d22f8864a21c8fd39e352ca8722e2726b90a789266a9d0fbdb81190e52e"}
Nov 03 17:56:45.205170 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:45.205167    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdf59d22f8864a21c8fd39e352ca8722e2726b90a789266a9d0fbdb81190e52e"
Nov 03 17:56:46.461060 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461018    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"]
Nov 03 17:56:46.461514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461499    4493 cpu_manager.go:414] "containerMap: removing container" podUID="385e15fc-41a0-4ccc-8129-82036ab6f8eb" containerName="revoke-keys"
Nov 03 17:56:46.461555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461515    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385e15fc-41a0-4ccc-8129-82036ab6f8eb" containerName="revoke-keys"
Nov 03 17:56:46.461555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461528    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8a3dcaed-470a-4a37-8193-95005dae91e1" containerName="revoke-keys"
Nov 03 17:56:46.461555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461533    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3dcaed-470a-4a37-8193-95005dae91e1" containerName="revoke-keys"
Nov 03 17:56:46.461555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461547    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2ff29423-9286-45fe-8b42-451d6117cdd8" containerName="revoke-keys"
Nov 03 17:56:46.461555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461552    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff29423-9286-45fe-8b42-451d6117cdd8" containerName="revoke-keys"
Nov 03 17:56:46.461704 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461622    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff29423-9286-45fe-8b42-451d6117cdd8" containerName="revoke-keys"
Nov 03 17:56:46.461704 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461634    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="385e15fc-41a0-4ccc-8129-82036ab6f8eb" containerName="revoke-keys"
Nov 03 17:56:46.461704 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.461640    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3dcaed-470a-4a37-8193-95005dae91e1" containerName="revoke-keys"
Nov 03 17:56:46.464903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.464879    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.497071 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.497045    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbcw\" (UniqueName: \"kubernetes.io/projected/40300b62-552c-47db-81fe-5e99065ae2d1-kube-api-access-tlbcw\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.497191 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.497175    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.497281 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.497268    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.524122 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.524095    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"]
Nov 03 17:56:46.571084 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.571055    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:56:46.571233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.571093    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:56:46.571233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.571094    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:56:46.571233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.571112    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:56:46.598132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.598104    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbcw\" (UniqueName: \"kubernetes.io/projected/40300b62-552c-47db-81fe-5e99065ae2d1-kube-api-access-tlbcw\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.598277 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.598186    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.598277 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.598240    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.598846 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.598821    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.600619 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.600602    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.662610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.662583    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbcw\" (UniqueName: \"kubernetes.io/projected/40300b62-552c-47db-81fe-5e99065ae2d1-kube-api-access-tlbcw\") pod \"maas-api-revoke-sub-cc65f59e-lkwhn\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:46.777401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:46.777360    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:47.165197 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:47.165167    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40300b62_552c_47db_81fe_5e99065ae2d1.slice/crio-14b86f1f51777f0e32d2b9b3cdf26a4e23f414c108f644f747c8c67766fb50fd WatchSource:0}: Error finding container 14b86f1f51777f0e32d2b9b3cdf26a4e23f414c108f644f747c8c67766fb50fd: Status 404 returned error can't find the container with id 14b86f1f51777f0e32d2b9b3cdf26a4e23f414c108f644f747c8c67766fb50fd
Nov 03 17:56:47.165910 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:47.165847    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"]
Nov 03 17:56:47.219263 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:47.219216    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn" event={"ID":"40300b62-552c-47db-81fe-5e99065ae2d1","Type":"ContainerStarted","Data":"14b86f1f51777f0e32d2b9b3cdf26a4e23f414c108f644f747c8c67766fb50fd"}
Nov 03 17:56:48.067273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:48.067236    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"]
Nov 03 17:56:48.117517 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:48.117481    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-629bf923-429zm"]
Nov 03 17:56:48.227016 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:48.226983    4493 generic.go:356] "Generic (PLEG): container finished" podID="40300b62-552c-47db-81fe-5e99065ae2d1" containerID="287d65364004310e7d3f32769dd09d5da89c70ea863d41d5eb5ceb41ea9d8042" exitCode=0
Nov 03 17:56:48.227199 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:48.227069    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn" event={"ID":"40300b62-552c-47db-81fe-5e99065ae2d1","Type":"ContainerDied","Data":"287d65364004310e7d3f32769dd09d5da89c70ea863d41d5eb5ceb41ea9d8042"}
Nov 03 17:56:48.712252 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:48.712217    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e841d75c-cff9-4ed7-9332-d40952b3813b" path="/var/lib/kubelet/pods/e841d75c-cff9-4ed7-9332-d40952b3813b/volumes"
Nov 03 17:56:49.394707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.394675    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:49.429926 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.429865    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-xks-ca\") pod \"40300b62-552c-47db-81fe-5e99065ae2d1\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") "
Nov 03 17:56:49.430074 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.429932    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-openshift-ca\") pod \"40300b62-552c-47db-81fe-5e99065ae2d1\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") "
Nov 03 17:56:49.430074 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.430048    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40300b62-552c-47db-81fe-5e99065ae2d1-kube-api-access-tlbcw\" (UniqueName: \"kubernetes.io/projected/40300b62-552c-47db-81fe-5e99065ae2d1-kube-api-access-tlbcw\") pod \"40300b62-552c-47db-81fe-5e99065ae2d1\" (UID: \"40300b62-552c-47db-81fe-5e99065ae2d1\") "
Nov 03 17:56:49.430279 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.430256    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-openshift-ca" pod "40300b62-552c-47db-81fe-5e99065ae2d1" (UID: "40300b62-552c-47db-81fe-5e99065ae2d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:56:49.430379 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.430366    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:49.432111 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.432090    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40300b62-552c-47db-81fe-5e99065ae2d1-kube-api-access-tlbcw" pod "40300b62-552c-47db-81fe-5e99065ae2d1" (UID: "40300b62-552c-47db-81fe-5e99065ae2d1"). InnerVolumeSpecName "kube-api-access-tlbcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:56:49.432200 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.432163    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-xks-ca" pod "40300b62-552c-47db-81fe-5e99065ae2d1" (UID: "40300b62-552c-47db-81fe-5e99065ae2d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:56:49.531802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.531730    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbcw\" (UniqueName: \"kubernetes.io/projected/40300b62-552c-47db-81fe-5e99065ae2d1-kube-api-access-tlbcw\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:49.531802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:49.531759    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40300b62-552c-47db-81fe-5e99065ae2d1-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:56:50.240737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:50.240707    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"
Nov 03 17:56:50.240737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:50.240738    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn" event={"ID":"40300b62-552c-47db-81fe-5e99065ae2d1","Type":"ContainerDied","Data":"14b86f1f51777f0e32d2b9b3cdf26a4e23f414c108f644f747c8c67766fb50fd"}
Nov 03 17:56:50.240975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:50.240787    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b86f1f51777f0e32d2b9b3cdf26a4e23f414c108f644f747c8c67766fb50fd"
Nov 03 17:56:51.063044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:51.062971    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"]
Nov 03 17:56:51.113775 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:51.113745    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e3e283b-bldmm"]
Nov 03 17:56:52.718563 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:52.718517    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e687f7-eac3-4317-949d-234a0bbdccb8" path="/var/lib/kubelet/pods/61e687f7-eac3-4317-949d-234a0bbdccb8/volumes"
Nov 03 17:56:54.076656 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.076620    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"]
Nov 03 17:56:54.077951 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.077925    4493 cpu_manager.go:414] "containerMap: removing container" podUID="40300b62-552c-47db-81fe-5e99065ae2d1" containerName="revoke-keys"
Nov 03 17:56:54.078105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.078088    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40300b62-552c-47db-81fe-5e99065ae2d1" containerName="revoke-keys"
Nov 03 17:56:54.078392 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.078374    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="40300b62-552c-47db-81fe-5e99065ae2d1" containerName="revoke-keys"
Nov 03 17:56:54.087176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.087149    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.199216 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.199169    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"]
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215073    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215159    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215188    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215265    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtxmp\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-kube-api-access-jtxmp\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215331    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215389    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215419    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215452    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.218430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.215563    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce03c0-8b18-424e-892d-ebde780fb409-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317339 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317306    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtxmp\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-kube-api-access-jtxmp\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317540 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317379    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317540 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317429    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317540 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317458    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317540 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317488    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317549    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce03c0-8b18-424e-892d-ebde780fb409-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317603    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317658    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.317740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.317685    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.318151 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.318127    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.318696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.318675    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.318969 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.318947    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.319393 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.319373    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.319632 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.319608    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce03c0-8b18-424e-892d-ebde780fb409-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.322474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.322454    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.323412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.323392    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.385963 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.385838    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.386203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.386175    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtxmp\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-kube-api-access-jtxmp\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.437204 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.437170    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:54.888960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.888934    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"]
Nov 03 17:56:54.889655 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:54.889628    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ce03c0_8b18_424e_892d_ebde780fb409.slice/crio-da52477d2d7ee1ac3a9223c3aa42ede188663b7fde3dd10db5cadaaf4293ca17 WatchSource:0}: Error finding container da52477d2d7ee1ac3a9223c3aa42ede188663b7fde3dd10db5cadaaf4293ca17: Status 404 returned error can't find the container with id da52477d2d7ee1ac3a9223c3aa42ede188663b7fde3dd10db5cadaaf4293ca17
Nov 03 17:56:54.892122 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.892091    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:54.892185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.892158    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:54.892225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:54.892205    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:55.301657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:55.301619    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" event={"ID":"a4ce03c0-8b18-424e-892d-ebde780fb409","Type":"ContainerStarted","Data":"219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51"}
Nov 03 17:56:55.302182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:55.301711    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" event={"ID":"a4ce03c0-8b18-424e-892d-ebde780fb409","Type":"ContainerStarted","Data":"da52477d2d7ee1ac3a9223c3aa42ede188663b7fde3dd10db5cadaaf4293ca17"}
Nov 03 17:56:55.429809 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:55.429754    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podStartSLOduration=1.429738932 podStartE2EDuration="1.429738932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:55.427464558 +0000 UTC m=+1394.886635546" watchObservedRunningTime="2026-11-03 17:56:55.429738932 +0000 UTC m=+1394.888909917"
Nov 03 17:56:55.438478 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:55.438438    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:56:55.441532 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:55.441018    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:56:55.441532 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:55.441118    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 17:56:56.045885 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.045826    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"]
Nov 03 17:56:56.049866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.049844    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.149949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.149912    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"]
Nov 03 17:56:56.160503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160469    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.160723 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160565    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.160723 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160648    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.160723 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160690    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pl74\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-kube-api-access-2pl74\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.160921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160754    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.160921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160858    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.161093 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160945    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.161093 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.160978    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.161093 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.161064    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.262627 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262592    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.262842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262690    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.262842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262713    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.262842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262760    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.262842 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262800    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pl74\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-kube-api-access-2pl74\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262858    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262907    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262944    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.262972    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263293 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.263131    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263293 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.263157    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263293 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.263209    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.263500    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.263890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.263853    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.266559 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.266535    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.271055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.271025    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.344942 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.343783    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.344942 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.343929    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pl74\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-kube-api-access-2pl74\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.365864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.365836    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:56.438898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.438308    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:56:56.438898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.438385    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 17:56:56.845688 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:56.845654    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c7bb05_062a_4e4c_b162_d54009ce9c2b.slice/crio-be9a2542aada33bb7d354c9b099bdde1498cf89e97bdc55f35388d3b7359c5d7 WatchSource:0}: Error finding container be9a2542aada33bb7d354c9b099bdde1498cf89e97bdc55f35388d3b7359c5d7: Status 404 returned error can't find the container with id be9a2542aada33bb7d354c9b099bdde1498cf89e97bdc55f35388d3b7359c5d7
Nov 03 17:56:56.845997 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.845973    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"]
Nov 03 17:56:56.848401 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.848363    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:56.848525 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.848443    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:56.848525 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:56.848482    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:56:57.348613 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.348575    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" event={"ID":"f4c7bb05-062a-4e4c-b162-d54009ce9c2b","Type":"ContainerStarted","Data":"db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783"}
Nov 03 17:56:57.349108 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.348636    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" event={"ID":"f4c7bb05-062a-4e4c-b162-d54009ce9c2b","Type":"ContainerStarted","Data":"be9a2542aada33bb7d354c9b099bdde1498cf89e97bdc55f35388d3b7359c5d7"}
Nov 03 17:56:57.366748 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.366721    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:56:57.368412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.368387    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:56:57.368541 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.368437    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 17:56:57.440337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.440304    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:56:57.440514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.440367    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 17:56:57.473266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.473213    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podStartSLOduration=2.473195336 podStartE2EDuration="2.473195336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:57.472613678 +0000 UTC m=+1396.931784665" watchObservedRunningTime="2026-11-03 17:56:57.473195336 +0000 UTC m=+1396.932366322"
Nov 03 17:56:57.892886 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.892842    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"]
Nov 03 17:56:57.896613 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.896598    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:57.969701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.968220    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:56:57.969701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.968321    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:56:57.969701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.968424    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:56:57.969701 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.968514    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:56:57.972659 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.971908    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"]
Nov 03 17:56:57.984769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.983457    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af2d765-7252-4f60-86de-939e5650e087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:57.984769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.983520    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af2d765-7252-4f60-86de-939e5650e087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:57.984769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:57.983748    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjm2g\" (UniqueName: \"kubernetes.io/projected/6af2d765-7252-4f60-86de-939e5650e087-kube-api-access-kjm2g\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.074066 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.074032    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"]
Nov 03 17:56:58.076765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.076745    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78815193-w7sh4"]
Nov 03 17:56:58.084412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.084387    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjm2g\" (UniqueName: \"kubernetes.io/projected/6af2d765-7252-4f60-86de-939e5650e087-kube-api-access-kjm2g\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.084554 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.084439    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af2d765-7252-4f60-86de-939e5650e087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.084554 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.084463    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af2d765-7252-4f60-86de-939e5650e087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.085118 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.085097    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af2d765-7252-4f60-86de-939e5650e087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.087231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.087212    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af2d765-7252-4f60-86de-939e5650e087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.166941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.166649    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjm2g\" (UniqueName: \"kubernetes.io/projected/6af2d765-7252-4f60-86de-939e5650e087-kube-api-access-kjm2g\") pod \"maas-api-revoke-sub-d2b70fc0-px8px\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.210627 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.210602    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:56:58.382903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.371851    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:56:58.382903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.371935    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 17:56:58.468901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.468297    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:56:58.468901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.468357    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 17:56:58.669237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.669196    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"]
Nov 03 17:56:58.671569 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:56:58.671538    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af2d765_7252_4f60_86de_939e5650e087.slice/crio-8889b8b0cf9327e37524c76fbe14801784073f04f3dab75e56be6336eea94a96 WatchSource:0}: Error finding container 8889b8b0cf9327e37524c76fbe14801784073f04f3dab75e56be6336eea94a96: Status 404 returned error can't find the container with id 8889b8b0cf9327e37524c76fbe14801784073f04f3dab75e56be6336eea94a96
Nov 03 17:56:58.712958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:58.712929    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ec2497-20d5-42bd-ad51-827fd0eb926c" path="/var/lib/kubelet/pods/82ec2497-20d5-42bd-ad51-827fd0eb926c/volumes"
Nov 03 17:56:59.066863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.066811    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"]
Nov 03 17:56:59.071627 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.071601    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3793979c-6548s"]
Nov 03 17:56:59.366765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.366317    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:56:59.366765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.366380    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 17:56:59.410035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.401348    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px" event={"ID":"6af2d765-7252-4f60-86de-939e5650e087","Type":"ContainerStarted","Data":"9ae84cc79a9fc91a36a388d8a4821a5e5609a93f0fabd8612a35e8360351522f"}
Nov 03 17:56:59.410035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.401426    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px" event={"ID":"6af2d765-7252-4f60-86de-939e5650e087","Type":"ContainerStarted","Data":"8889b8b0cf9327e37524c76fbe14801784073f04f3dab75e56be6336eea94a96"}
Nov 03 17:56:59.438900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.438202    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:56:59.438900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.438259    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 17:56:59.539899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:56:59.538639    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px" podStartSLOduration=2.53861831 podStartE2EDuration="2.53861831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:59.536795463 +0000 UTC m=+1398.995966462" watchObservedRunningTime="2026-11-03 17:56:59.53861831 +0000 UTC m=+1398.997789298"
Nov 03 17:57:00.366471 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:00.366436    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:00.366772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:00.366743    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 17:57:00.410605 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:00.410573    4493 generic.go:356] "Generic (PLEG): container finished" podID="6af2d765-7252-4f60-86de-939e5650e087" containerID="9ae84cc79a9fc91a36a388d8a4821a5e5609a93f0fabd8612a35e8360351522f" exitCode=0
Nov 03 17:57:00.411197 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:00.410650    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px" event={"ID":"6af2d765-7252-4f60-86de-939e5650e087","Type":"ContainerDied","Data":"9ae84cc79a9fc91a36a388d8a4821a5e5609a93f0fabd8612a35e8360351522f"}
Nov 03 17:57:00.448325 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:00.448284    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:00.448622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:00.448592    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 17:57:00.713010 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:00.712985    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56402ba4-5419-40d2-b92e-2cf682432625" path="/var/lib/kubelet/pods/56402ba4-5419-40d2-b92e-2cf682432625/volumes"
Nov 03 17:57:01.366761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.366729    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:01.366975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.366793    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 17:57:01.438899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.438255    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 17:57:01.438899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.438312    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 17:57:01.653307 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.653248    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:57:01.775246 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.775202    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af2d765-7252-4f60-86de-939e5650e087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af2d765-7252-4f60-86de-939e5650e087-maas-api-openshift-ca\") pod \"6af2d765-7252-4f60-86de-939e5650e087\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") "
Nov 03 17:57:01.775440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.775401    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af2d765-7252-4f60-86de-939e5650e087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af2d765-7252-4f60-86de-939e5650e087-maas-api-xks-ca\") pod \"6af2d765-7252-4f60-86de-939e5650e087\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") "
Nov 03 17:57:01.775524 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.775469    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af2d765-7252-4f60-86de-939e5650e087-kube-api-access-kjm2g\" (UniqueName: \"kubernetes.io/projected/6af2d765-7252-4f60-86de-939e5650e087-kube-api-access-kjm2g\") pod \"6af2d765-7252-4f60-86de-939e5650e087\" (UID: \"6af2d765-7252-4f60-86de-939e5650e087\") "
Nov 03 17:57:01.776560 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.776515    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af2d765-7252-4f60-86de-939e5650e087-maas-api-openshift-ca" pod "6af2d765-7252-4f60-86de-939e5650e087" (UID: "6af2d765-7252-4f60-86de-939e5650e087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:01.779043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.778992    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af2d765-7252-4f60-86de-939e5650e087-maas-api-xks-ca" pod "6af2d765-7252-4f60-86de-939e5650e087" (UID: "6af2d765-7252-4f60-86de-939e5650e087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:01.780911 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.780284    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af2d765-7252-4f60-86de-939e5650e087-kube-api-access-kjm2g" pod "6af2d765-7252-4f60-86de-939e5650e087" (UID: "6af2d765-7252-4f60-86de-939e5650e087"). InnerVolumeSpecName "kube-api-access-kjm2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:01.876929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.876893    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjm2g\" (UniqueName: \"kubernetes.io/projected/6af2d765-7252-4f60-86de-939e5650e087-kube-api-access-kjm2g\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:01.876929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.876923    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af2d765-7252-4f60-86de-939e5650e087-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:01.877121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.876939    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af2d765-7252-4f60-86de-939e5650e087-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:01.945506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.945390    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"]
Nov 03 17:57:01.946498 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.946467    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6af2d765-7252-4f60-86de-939e5650e087" containerName="revoke-keys"
Nov 03 17:57:01.946498 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.946496    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af2d765-7252-4f60-86de-939e5650e087" containerName="revoke-keys"
Nov 03 17:57:01.946658 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.946626    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af2d765-7252-4f60-86de-939e5650e087" containerName="revoke-keys"
Nov 03 17:57:01.950608 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:01.950588    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.000861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.000830    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"]
Nov 03 17:57:02.078542 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.078505    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.078737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.078547    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zns\" (UniqueName: \"kubernetes.io/projected/45430aec-4f43-4d58-b907-80f0312db35e-kube-api-access-c4zns\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.078737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.078625    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.179561 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.179524    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.179769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.179573    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zns\" (UniqueName: \"kubernetes.io/projected/45430aec-4f43-4d58-b907-80f0312db35e-kube-api-access-c4zns\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.179769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.179608    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.180474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.180453    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.182294 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.182273    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.246836 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.246801    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zns\" (UniqueName: \"kubernetes.io/projected/45430aec-4f43-4d58-b907-80f0312db35e-kube-api-access-c4zns\") pod \"maas-api-revoke-sub-3fca5c3d-g2nff\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.267447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.267419    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:02.368857 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.367791    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:02.368857 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.367849    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 17:57:02.393951 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.392207    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"]
Nov 03 17:57:02.404015 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.402246    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.465793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.463312    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"]
Nov 03 17:57:02.467712 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.467684    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"
Nov 03 17:57:02.467847 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.467683    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px" event={"ID":"6af2d765-7252-4f60-86de-939e5650e087","Type":"ContainerDied","Data":"8889b8b0cf9327e37524c76fbe14801784073f04f3dab75e56be6336eea94a96"}
Nov 03 17:57:02.467944 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.467856    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8889b8b0cf9327e37524c76fbe14801784073f04f3dab75e56be6336eea94a96"
Nov 03 17:57:02.494114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.494036    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.494114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.494088    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.494313 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.494233    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtq62\" (UniqueName: \"kubernetes.io/projected/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-kube-api-access-xtq62\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.595937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.595831    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.595937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.595927    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.596161 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.596057    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtq62\" (UniqueName: \"kubernetes.io/projected/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-kube-api-access-xtq62\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.596598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.596560    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.598841 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.598813    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.647899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.647236    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:57:02.648739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.648716    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:57:02.652163 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.652143    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 17:57:02.659857 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.659837    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtq62\" (UniqueName: \"kubernetes.io/projected/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-kube-api-access-xtq62\") pod \"maas-api-revoke-sub-766a9c5e-k47bs\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:02.701022 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.700997    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"]
Nov 03 17:57:02.701860 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:02.701837    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45430aec_4f43_4d58_b907_80f0312db35e.slice/crio-d74fb3d768ef020d274697039c2782b97b8e6d7c9a47c3796e9e00e3e239fa59 WatchSource:0}: Error finding container d74fb3d768ef020d274697039c2782b97b8e6d7c9a47c3796e9e00e3e239fa59: Status 404 returned error can't find the container with id d74fb3d768ef020d274697039c2782b97b8e6d7c9a47c3796e9e00e3e239fa59
Nov 03 17:57:02.767986 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:02.767954    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:03.177557 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.177526    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"]
Nov 03 17:57:03.179340 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:03.179309    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad6d457_21d9_45ad_b594_b5cfbcb496d2.slice/crio-aeb46b0f605dac3c6ccc8a8982faba6de5c707703b21e0a71bde87d8a3803b9f WatchSource:0}: Error finding container aeb46b0f605dac3c6ccc8a8982faba6de5c707703b21e0a71bde87d8a3803b9f: Status 404 returned error can't find the container with id aeb46b0f605dac3c6ccc8a8982faba6de5c707703b21e0a71bde87d8a3803b9f
Nov 03 17:57:03.367566 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.367537    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 17:57:03.367694 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.367588    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 17:57:03.476544 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.476506    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs" event={"ID":"5ad6d457-21d9-45ad-b594-b5cfbcb496d2","Type":"ContainerStarted","Data":"92ab83ec2712b31586a00bee18e7b2c9d26477c227fbcd2c1decc534fdeff505"}
Nov 03 17:57:03.476966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.476578    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs" event={"ID":"5ad6d457-21d9-45ad-b594-b5cfbcb496d2","Type":"ContainerStarted","Data":"aeb46b0f605dac3c6ccc8a8982faba6de5c707703b21e0a71bde87d8a3803b9f"}
Nov 03 17:57:03.478299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.478275    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff" event={"ID":"45430aec-4f43-4d58-b907-80f0312db35e","Type":"ContainerStarted","Data":"ee34b46c05e34dce936f274873b29b4db52c4455cf2b6b831552106e3fedb614"}
Nov 03 17:57:03.478420 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.478317    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff" event={"ID":"45430aec-4f43-4d58-b907-80f0312db35e","Type":"ContainerStarted","Data":"d74fb3d768ef020d274697039c2782b97b8e6d7c9a47c3796e9e00e3e239fa59"}
Nov 03 17:57:03.601903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.601817    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs" podStartSLOduration=1.601797014 podStartE2EDuration="1.601797014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:03.597864423 +0000 UTC m=+1403.057035410" watchObservedRunningTime="2026-11-03 17:57:03.601797014 +0000 UTC m=+1403.060968002"
Nov 03 17:57:03.720621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:03.720563    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff" podStartSLOduration=2.720549472 podStartE2EDuration="2.720549472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:03.718230852 +0000 UTC m=+1403.177401839" watchObservedRunningTime="2026-11-03 17:57:03.720549472 +0000 UTC m=+1403.179720457"
Nov 03 17:57:04.076677 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.076630    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"]
Nov 03 17:57:04.081852 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.081820    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c27b3b3-4zf6n"]
Nov 03 17:57:04.408955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.408844    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:57:04.409274 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.409253    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:57:04.409988 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.409967    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 17:57:04.487601 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.487569    4493 generic.go:356] "Generic (PLEG): container finished" podID="45430aec-4f43-4d58-b907-80f0312db35e" containerID="ee34b46c05e34dce936f274873b29b4db52c4455cf2b6b831552106e3fedb614" exitCode=0
Nov 03 17:57:04.488024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.487654    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff" event={"ID":"45430aec-4f43-4d58-b907-80f0312db35e","Type":"ContainerDied","Data":"ee34b46c05e34dce936f274873b29b4db52c4455cf2b6b831552106e3fedb614"}
Nov 03 17:57:04.489439 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.489413    4493 generic.go:356] "Generic (PLEG): container finished" podID="5ad6d457-21d9-45ad-b594-b5cfbcb496d2" containerID="92ab83ec2712b31586a00bee18e7b2c9d26477c227fbcd2c1decc534fdeff505" exitCode=0
Nov 03 17:57:04.489582 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.489508    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs" event={"ID":"5ad6d457-21d9-45ad-b594-b5cfbcb496d2","Type":"ContainerDied","Data":"92ab83ec2712b31586a00bee18e7b2c9d26477c227fbcd2c1decc534fdeff505"}
Nov 03 17:57:04.722245 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:04.722209    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4f867a-f1a7-4685-84c6-67ad1499ef31" path="/var/lib/kubelet/pods/ba4f867a-f1a7-4685-84c6-67ad1499ef31/volumes"
Nov 03 17:57:05.292421 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.292388    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"]
Nov 03 17:57:05.296308 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.296291    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.328650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.328619    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.328757 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.328664    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.328757 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.328743    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/34caf5ab-53b1-4b57-928f-8d056efea5ea-kube-api-access-k2f52\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.355641 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.355612    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"]
Nov 03 17:57:05.429620 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.429587    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.429757 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.429634    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.429757 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.429673    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/34caf5ab-53b1-4b57-928f-8d056efea5ea-kube-api-access-k2f52\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.430407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.430328    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.432105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.432083    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.493797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.493773    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/34caf5ab-53b1-4b57-928f-8d056efea5ea-kube-api-access-k2f52\") pod \"maas-api-revoke-sub-a5697d93-xttgb\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.609802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.609744    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:05.655650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.655609    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:05.687764 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.687412    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:05.733598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.733572    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-openshift-ca\") pod \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") "
Nov 03 17:57:05.733727 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.733629    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-xks-ca\") pod \"45430aec-4f43-4d58-b907-80f0312db35e\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") "
Nov 03 17:57:05.733727 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.733669    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-kube-api-access-xtq62\" (UniqueName: \"kubernetes.io/projected/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-kube-api-access-xtq62\") pod \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") "
Nov 03 17:57:05.733897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.733739    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45430aec-4f43-4d58-b907-80f0312db35e-kube-api-access-c4zns\" (UniqueName: \"kubernetes.io/projected/45430aec-4f43-4d58-b907-80f0312db35e-kube-api-access-c4zns\") pod \"45430aec-4f43-4d58-b907-80f0312db35e\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") "
Nov 03 17:57:05.733897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.733844    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-openshift-ca\") pod \"45430aec-4f43-4d58-b907-80f0312db35e\" (UID: \"45430aec-4f43-4d58-b907-80f0312db35e\") "
Nov 03 17:57:05.733897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.733889    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-xks-ca\") pod \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\" (UID: \"5ad6d457-21d9-45ad-b594-b5cfbcb496d2\") "
Nov 03 17:57:05.734041 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.733994    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-openshift-ca" pod "5ad6d457-21d9-45ad-b594-b5cfbcb496d2" (UID: "5ad6d457-21d9-45ad-b594-b5cfbcb496d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:05.734338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.734309    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-openshift-ca" pod "45430aec-4f43-4d58-b907-80f0312db35e" (UID: "45430aec-4f43-4d58-b907-80f0312db35e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:05.734494 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.734469    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.734622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.734498    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.736184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.736158    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45430aec-4f43-4d58-b907-80f0312db35e-kube-api-access-c4zns" pod "45430aec-4f43-4d58-b907-80f0312db35e" (UID: "45430aec-4f43-4d58-b907-80f0312db35e"). InnerVolumeSpecName "kube-api-access-c4zns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:05.736184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.736173    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-xks-ca" pod "5ad6d457-21d9-45ad-b594-b5cfbcb496d2" (UID: "5ad6d457-21d9-45ad-b594-b5cfbcb496d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:05.736426 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.736411    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-kube-api-access-xtq62" pod "5ad6d457-21d9-45ad-b594-b5cfbcb496d2" (UID: "5ad6d457-21d9-45ad-b594-b5cfbcb496d2"). InnerVolumeSpecName "kube-api-access-xtq62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:05.736748 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.736735    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-xks-ca" pod "45430aec-4f43-4d58-b907-80f0312db35e" (UID: "45430aec-4f43-4d58-b907-80f0312db35e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:05.836204 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.836127    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.836204 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.836152    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45430aec-4f43-4d58-b907-80f0312db35e-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.836204 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.836163    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtq62\" (UniqueName: \"kubernetes.io/projected/5ad6d457-21d9-45ad-b594-b5cfbcb496d2-kube-api-access-xtq62\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.836204 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.836173    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4zns\" (UniqueName: \"kubernetes.io/projected/45430aec-4f43-4d58-b907-80f0312db35e-kube-api-access-c4zns\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:05.955690 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.955654    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"]
Nov 03 17:57:05.956177 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.956163    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5ad6d457-21d9-45ad-b594-b5cfbcb496d2" containerName="revoke-keys"
Nov 03 17:57:05.956246 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.956178    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad6d457-21d9-45ad-b594-b5cfbcb496d2" containerName="revoke-keys"
Nov 03 17:57:05.956246 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.956204    4493 cpu_manager.go:414] "containerMap: removing container" podUID="45430aec-4f43-4d58-b907-80f0312db35e" containerName="revoke-keys"
Nov 03 17:57:05.956246 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.956212    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45430aec-4f43-4d58-b907-80f0312db35e" containerName="revoke-keys"
Nov 03 17:57:05.956337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.956303    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="45430aec-4f43-4d58-b907-80f0312db35e" containerName="revoke-keys"
Nov 03 17:57:05.956337 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.956314    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad6d457-21d9-45ad-b594-b5cfbcb496d2" containerName="revoke-keys"
Nov 03 17:57:05.959770 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:05.959750    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.004470 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.004436    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"]
Nov 03 17:57:06.004713 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:06.004688    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34caf5ab_53b1_4b57_928f_8d056efea5ea.slice/crio-1f05d3efd6d90812b54878bcdf756a2f325bf5e445c15ae6fa362039bb060ac6 WatchSource:0}: Error finding container 1f05d3efd6d90812b54878bcdf756a2f325bf5e445c15ae6fa362039bb060ac6: Status 404 returned error can't find the container with id 1f05d3efd6d90812b54878bcdf756a2f325bf5e445c15ae6fa362039bb060ac6
Nov 03 17:57:06.023213 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.023190    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"]
Nov 03 17:57:06.038451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.038414    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/50b425fc-14ba-440e-a596-310a9261dec8-kube-api-access-xq9bg\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.038556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.038453    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.038556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.038476    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.139633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.139562    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/50b425fc-14ba-440e-a596-310a9261dec8-kube-api-access-xq9bg\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.139633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.139605    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.139861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.139839    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.140246 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.140226    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.142185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.142165    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.206784 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.206756    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/50b425fc-14ba-440e-a596-310a9261dec8-kube-api-access-xq9bg\") pod \"maas-api-revoke-sub-aa9226da-rttbm\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.275835 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.275807    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:06.506442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.506418    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"
Nov 03 17:57:06.506868 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.506418    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff" event={"ID":"45430aec-4f43-4d58-b907-80f0312db35e","Type":"ContainerDied","Data":"d74fb3d768ef020d274697039c2782b97b8e6d7c9a47c3796e9e00e3e239fa59"}
Nov 03 17:57:06.506868 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.506584    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d74fb3d768ef020d274697039c2782b97b8e6d7c9a47c3796e9e00e3e239fa59"
Nov 03 17:57:06.507903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.507855    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"
Nov 03 17:57:06.508009 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.507896    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs" event={"ID":"5ad6d457-21d9-45ad-b594-b5cfbcb496d2","Type":"ContainerDied","Data":"aeb46b0f605dac3c6ccc8a8982faba6de5c707703b21e0a71bde87d8a3803b9f"}
Nov 03 17:57:06.508009 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.507945    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeb46b0f605dac3c6ccc8a8982faba6de5c707703b21e0a71bde87d8a3803b9f"
Nov 03 17:57:06.509569 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.509543    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb" event={"ID":"34caf5ab-53b1-4b57-928f-8d056efea5ea","Type":"ContainerStarted","Data":"5fc0d0ba39e86f879194517f7fa8c37c776b5dfae40e1f9a5d7c1e38dd3ba50d"}
Nov 03 17:57:06.509646 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.509586    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb" event={"ID":"34caf5ab-53b1-4b57-928f-8d056efea5ea","Type":"ContainerStarted","Data":"1f05d3efd6d90812b54878bcdf756a2f325bf5e445c15ae6fa362039bb060ac6"}
Nov 03 17:57:06.630660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.630616    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb" podStartSLOduration=1.630602576 podStartE2EDuration="1.630602576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:06.628831779 +0000 UTC m=+1406.088002761" watchObservedRunningTime="2026-11-03 17:57:06.630602576 +0000 UTC m=+1406.089773562"
Nov 03 17:57:06.660698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:06.660677    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"]
Nov 03 17:57:06.661786 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:06.661762    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b425fc_14ba_440e_a596_310a9261dec8.slice/crio-d7b40f654d093beadbdffbc4ad23f0d079f438d082ec83a455342d8e6f732948 WatchSource:0}: Error finding container d7b40f654d093beadbdffbc4ad23f0d079f438d082ec83a455342d8e6f732948: Status 404 returned error can't find the container with id d7b40f654d093beadbdffbc4ad23f0d079f438d082ec83a455342d8e6f732948
Nov 03 17:57:06.994712 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:57:06.994673    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:57:07.058646 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.058614    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"]
Nov 03 17:57:07.062824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.062800    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e47aeb6-f8d6v"]
Nov 03 17:57:07.518683 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.518646    4493 generic.go:356] "Generic (PLEG): container finished" podID="34caf5ab-53b1-4b57-928f-8d056efea5ea" containerID="5fc0d0ba39e86f879194517f7fa8c37c776b5dfae40e1f9a5d7c1e38dd3ba50d" exitCode=0
Nov 03 17:57:07.519121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.518737    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb" event={"ID":"34caf5ab-53b1-4b57-928f-8d056efea5ea","Type":"ContainerDied","Data":"5fc0d0ba39e86f879194517f7fa8c37c776b5dfae40e1f9a5d7c1e38dd3ba50d"}
Nov 03 17:57:07.520698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.520669    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm" event={"ID":"50b425fc-14ba-440e-a596-310a9261dec8","Type":"ContainerStarted","Data":"fafd3b4e924262db6d126bccbed73a1ae2d4da0e10fee8442d819685ef50c59c"}
Nov 03 17:57:07.520820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.520721    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm" event={"ID":"50b425fc-14ba-440e-a596-310a9261dec8","Type":"ContainerStarted","Data":"d7b40f654d093beadbdffbc4ad23f0d079f438d082ec83a455342d8e6f732948"}
Nov 03 17:57:07.759850 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.759781    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm" podStartSLOduration=2.759767252 podStartE2EDuration="2.759767252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:07.757843326 +0000 UTC m=+1407.217014315" watchObservedRunningTime="2026-11-03 17:57:07.759767252 +0000 UTC m=+1407.218938238"
Nov 03 17:57:07.880069 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.879993    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"]
Nov 03 17:57:07.884105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.884083    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:07.943845 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.943811    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"]
Nov 03 17:57:07.964329 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.964289    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:07.964488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.964469    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:07.964547 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:07.964529    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njllm\" (UniqueName: \"kubernetes.io/projected/511d605d-a149-4ab6-a6a2-07c267a474b3-kube-api-access-njllm\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.060818 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.060786    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"]
Nov 03 17:57:08.065445 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.065416    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.065598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.065501    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.065598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.065538    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njllm\" (UniqueName: \"kubernetes.io/projected/511d605d-a149-4ab6-a6a2-07c267a474b3-kube-api-access-njllm\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.065989 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.065968    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f082c984-stxff"]
Nov 03 17:57:08.066163 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.066141    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.068044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.068028    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.129379 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.129342    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njllm\" (UniqueName: \"kubernetes.io/projected/511d605d-a149-4ab6-a6a2-07c267a474b3-kube-api-access-njllm\") pod \"maas-api-revoke-sub-0b54c9df-wb5k4\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.197169 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.197090    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:08.443005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.442968    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"]
Nov 03 17:57:08.447987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.447933    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471144 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471114    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471154    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471235    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471360 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471288    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471360 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471343    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471489    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471541 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471524    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471581 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471559    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxfx\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-kube-api-access-jrxfx\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.471638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.471624    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.497046 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.497015    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"]
Nov 03 17:57:08.500967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.500947    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.515691 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.515663    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"]
Nov 03 17:57:08.529866 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.529828    4493 generic.go:356] "Generic (PLEG): container finished" podID="50b425fc-14ba-440e-a596-310a9261dec8" containerID="fafd3b4e924262db6d126bccbed73a1ae2d4da0e10fee8442d819685ef50c59c" exitCode=0
Nov 03 17:57:08.530307 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.529914    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm" event={"ID":"50b425fc-14ba-440e-a596-310a9261dec8","Type":"ContainerDied","Data":"fafd3b4e924262db6d126bccbed73a1ae2d4da0e10fee8442d819685ef50c59c"}
Nov 03 17:57:08.573229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573189    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.573400 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573294    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.573454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573416    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.573488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573451    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.573519 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573500    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.573565 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573549    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxfx\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-kube-api-access-jrxfx\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.573617 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573589    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.573679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573621    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.573728 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573677    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf94c4a-b570-4670-942c-8e981619c397-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.573728 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573710    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.573839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573748    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.573839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573772    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.573839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573811    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb26w\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-kube-api-access-sb26w\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.573839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573823    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.574102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573844    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.574102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573895    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.574102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.573976    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf94c4a-b570-4670-942c-8e981619c397-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.574344 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.574318    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.574472 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.574342    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.574472 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.574389    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.574574 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.574544    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.574779 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.574678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.574991 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.574969    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.579130 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.579100    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.584943 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.582638    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.588248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.588220    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"]
Nov 03 17:57:08.594384 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.594356    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.615798 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.615740    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"]
Nov 03 17:57:08.624164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.624136    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"]
Nov 03 17:57:08.644745 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.644715    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.665363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.665324    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"]
Nov 03 17:57:08.666747 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.666617    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxfx\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-kube-api-access-jrxfx\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.675733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.675704    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf94c4a-b570-4670-942c-8e981619c397-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.675906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.675774    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.675906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.675801    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.675906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.675838    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb26w\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-kube-api-access-sb26w\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.675906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.675883    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.676132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.675940    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf94c4a-b570-4670-942c-8e981619c397-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.676132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.675978    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.676132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.676102    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca63299-6b3b-4242-9a3d-022fea22c541-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.676461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.676435    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.676574 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.676504    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.676574 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.676556    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.676677 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.676616    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.676677 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.676659    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.676941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.676912    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.677203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.677173    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.677423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.677398    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf94c4a-b570-4670-942c-8e981619c397-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.677686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.677662    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.677983 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.677957    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.678104 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.678027    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7nk\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-kube-api-access-lz7nk\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.678176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.678117    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.678235 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.678186    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca63299-6b3b-4242-9a3d-022fea22c541-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.678288 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.678271    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.678639 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.678615    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.680188 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.680101    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.680745 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.680721    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf94c4a-b570-4670-942c-8e981619c397-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.715154 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.715116    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:08.731284 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.731244    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb97f7f-8c04-45b3-b97b-cf4607409e74" path="/var/lib/kubelet/pods/5cb97f7f-8c04-45b3-b97b-cf4607409e74/volumes"
Nov 03 17:57:08.731959 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.731939    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777a3204-e98b-45c0-af1c-30b622f244ec" path="/var/lib/kubelet/pods/777a3204-e98b-45c0-af1c-30b622f244ec/volumes"
Nov 03 17:57:08.762499 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.762462    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:08.778907 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.778858    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-openshift-ca\") pod \"34caf5ab-53b1-4b57-928f-8d056efea5ea\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") "
Nov 03 17:57:08.779033 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.778999    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-xks-ca\") pod \"34caf5ab-53b1-4b57-928f-8d056efea5ea\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") "
Nov 03 17:57:08.779084 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779061    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34caf5ab-53b1-4b57-928f-8d056efea5ea-kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/34caf5ab-53b1-4b57-928f-8d056efea5ea-kube-api-access-k2f52\") pod \"34caf5ab-53b1-4b57-928f-8d056efea5ea\" (UID: \"34caf5ab-53b1-4b57-928f-8d056efea5ea\") "
Nov 03 17:57:08.779229 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779208    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-openshift-ca" pod "34caf5ab-53b1-4b57-928f-8d056efea5ea" (UID: "34caf5ab-53b1-4b57-928f-8d056efea5ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:08.779328 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779309    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.779387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779372    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca63299-6b3b-4242-9a3d-022fea22c541-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.779443 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779430    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.779520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779506    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779652    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779704    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7nk\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-kube-api-access-lz7nk\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779752    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779769    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca63299-6b3b-4242-9a3d-022fea22c541-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779822    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779925    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.779934    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.780191    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.780404    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.780513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.780442    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.781131 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.781096    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca63299-6b3b-4242-9a3d-022fea22c541-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.816043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.815998    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34caf5ab-53b1-4b57-928f-8d056efea5ea-kube-api-access-k2f52" pod "34caf5ab-53b1-4b57-928f-8d056efea5ea" (UID: "34caf5ab-53b1-4b57-928f-8d056efea5ea"). InnerVolumeSpecName "kube-api-access-k2f52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:08.816439 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.816415    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca63299-6b3b-4242-9a3d-022fea22c541-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.816539 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.816449    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.816615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.816594    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-xks-ca" pod "34caf5ab-53b1-4b57-928f-8d056efea5ea" (UID: "34caf5ab-53b1-4b57-928f-8d056efea5ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:08.816704 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.816687    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.816917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.816897    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb26w\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-kube-api-access-sb26w\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:08.835625 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.835596    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"]
Nov 03 17:57:08.836352 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.836330    4493 cpu_manager.go:414] "containerMap: removing container" podUID="34caf5ab-53b1-4b57-928f-8d056efea5ea" containerName="revoke-keys"
Nov 03 17:57:08.836465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.836353    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34caf5ab-53b1-4b57-928f-8d056efea5ea" containerName="revoke-keys"
Nov 03 17:57:08.836507 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.836480    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="34caf5ab-53b1-4b57-928f-8d056efea5ea" containerName="revoke-keys"
Nov 03 17:57:08.843150 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.843129    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.847696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.847670    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.852022 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.851992    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7nk\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-kube-api-access-lz7nk\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.881994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.880033    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"]
Nov 03 17:57:08.888906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.888813    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.892815 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.892782    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.892916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.892833    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.892954 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.892914    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.892954 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.892938    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.893031 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.892967    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.893075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.893057    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.893131 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.893117    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.893609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.893189    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.893609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.893434    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-kube-api-access-wdz7b\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.893609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.893546    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34caf5ab-53b1-4b57-928f-8d056efea5ea-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:08.893609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.893563    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2f52\" (UniqueName: \"kubernetes.io/projected/34caf5ab-53b1-4b57-928f-8d056efea5ea-kube-api-access-k2f52\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:08.905146 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.904134    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"]
Nov 03 17:57:08.917935 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.917524    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:08.941230 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.941203    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"]
Nov 03 17:57:08.995290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995258    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.995441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995312    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.995441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995349    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.995441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995387    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.995441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995412    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.995679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995650    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.995740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995709    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.995740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995729    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-kube-api-access-wdz7b\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.995836 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995784    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.995836 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995824    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.995956 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995849    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.995956 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.995886    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.996103 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996039    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.996258 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996243    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.996298 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996283    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.996342 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996312    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc030dce-3454-4c6c-8ebb-e532e89676ef-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.996386 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996339    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.996386 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996364    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.996460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996408    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.997240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996563    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.997240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996620    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fp5\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-kube-api-access-h2fp5\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:08.997240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.996651    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.997240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.997201    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.997950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.997932    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:08.998584 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:08.998569    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:09.059720 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.059686    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:09.060022 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.059997    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-kube-api-access-wdz7b\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:09.098151 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098115    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098306 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098193    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098306 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098264    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098420 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098321    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098420 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098358    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098420 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098384    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098467    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098491    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc030dce-3454-4c6c-8ebb-e532e89676ef-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098555    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fp5\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-kube-api-access-h2fp5\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.098815 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.098679    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.099205 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.099181    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.099323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.099231    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.099323 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.099300    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc030dce-3454-4c6c-8ebb-e532e89676ef-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.099465 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.099439    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.100836 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.100814    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.101450 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.101428    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.114595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.114575    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:09.160006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.159972    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:09.164742 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.164701    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fp5\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-kube-api-access-h2fp5\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.164944 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.164921    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.206684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.206647    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"]
Nov 03 17:57:09.211592 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.211337    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:09.213834 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.213793    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.213941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.213913    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.213979 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.213965    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.354628 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.354600    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"]
Nov 03 17:57:09.355226 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:09.355202    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca63299_6b3b_4242_9a3d_022fea22c541.slice/crio-300c54f7c4ac29b6df05685707e6cfe0a972fd15f3d59fa87632cbbeda69322b WatchSource:0}: Error finding container 300c54f7c4ac29b6df05685707e6cfe0a972fd15f3d59fa87632cbbeda69322b: Status 404 returned error can't find the container with id 300c54f7c4ac29b6df05685707e6cfe0a972fd15f3d59fa87632cbbeda69322b
Nov 03 17:57:09.358334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.358293    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.358447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.358368    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.358447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.358397    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.515810 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.515779    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"]
Nov 03 17:57:09.516832 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:09.516800    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf94c4a_b570_4670_942c_8e981619c397.slice/crio-7f73dbd8d4596102f7e9580b00decce35a8b3bce7e5bccb9571be7b86153bf57 WatchSource:0}: Error finding container 7f73dbd8d4596102f7e9580b00decce35a8b3bce7e5bccb9571be7b86153bf57: Status 404 returned error can't find the container with id 7f73dbd8d4596102f7e9580b00decce35a8b3bce7e5bccb9571be7b86153bf57
Nov 03 17:57:09.521023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.520978    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.521124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.521072    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.521124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.521117    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.544060 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.544030    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" event={"ID":"8cf94c4a-b570-4670-942c-8e981619c397","Type":"ContainerStarted","Data":"7f73dbd8d4596102f7e9580b00decce35a8b3bce7e5bccb9571be7b86153bf57"}
Nov 03 17:57:09.551417 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.551388    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb" event={"ID":"34caf5ab-53b1-4b57-928f-8d056efea5ea","Type":"ContainerDied","Data":"1f05d3efd6d90812b54878bcdf756a2f325bf5e445c15ae6fa362039bb060ac6"}
Nov 03 17:57:09.551531 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.551418    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"
Nov 03 17:57:09.551531 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.551440    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f05d3efd6d90812b54878bcdf756a2f325bf5e445c15ae6fa362039bb060ac6"
Nov 03 17:57:09.557261 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.557228    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4" event={"ID":"511d605d-a149-4ab6-a6a2-07c267a474b3","Type":"ContainerStarted","Data":"7a78b5baffec4a3195f0842aacb91b6d6a9ceedb236ef17b4871dbf1067d9212"}
Nov 03 17:57:09.557406 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.557289    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4" event={"ID":"511d605d-a149-4ab6-a6a2-07c267a474b3","Type":"ContainerStarted","Data":"a7de60bb3d5d231364445cf164b605b365db61939b29a513785e2987e0ea287e"}
Nov 03 17:57:09.562106 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.561585    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" event={"ID":"eca63299-6b3b-4242-9a3d-022fea22c541","Type":"ContainerStarted","Data":"bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e"}
Nov 03 17:57:09.562106 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.561625    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" event={"ID":"eca63299-6b3b-4242-9a3d-022fea22c541","Type":"ContainerStarted","Data":"300c54f7c4ac29b6df05685707e6cfe0a972fd15f3d59fa87632cbbeda69322b"}
Nov 03 17:57:09.563569 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.563547    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" event={"ID":"1f7c15d7-8dcb-4644-9c6e-cade90d240e4","Type":"ContainerStarted","Data":"45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a"}
Nov 03 17:57:09.563650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.563589    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" event={"ID":"1f7c15d7-8dcb-4644-9c6e-cade90d240e4","Type":"ContainerStarted","Data":"92896bf1687f27d7922ebf994b03af9817e03d3e00aff082d57cb0155caa8220"}
Nov 03 17:57:09.573304 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.572949    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"]
Nov 03 17:57:09.573606 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:09.573573    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c84f63_1f56_4d39_af09_ee4a3b8f35e1.slice/crio-5cbcaf843b46d8cbfb3f7cf08063b861e68b130fa15b2b6088f5ef7cae0abb3b WatchSource:0}: Error finding container 5cbcaf843b46d8cbfb3f7cf08063b861e68b130fa15b2b6088f5ef7cae0abb3b: Status 404 returned error can't find the container with id 5cbcaf843b46d8cbfb3f7cf08063b861e68b130fa15b2b6088f5ef7cae0abb3b
Nov 03 17:57:09.577974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.577448    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.577974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.577529    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.577974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.577577    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.640195 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.640166    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"]
Nov 03 17:57:09.642615 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:09.642575    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc030dce_3454_4c6c_8ebb_e532e89676ef.slice/crio-3ee7eb17d3a63daee094c49925f0fc8e74cb1df212ef5d8c5a15da6eab965c48 WatchSource:0}: Error finding container 3ee7eb17d3a63daee094c49925f0fc8e74cb1df212ef5d8c5a15da6eab965c48: Status 404 returned error can't find the container with id 3ee7eb17d3a63daee094c49925f0fc8e74cb1df212ef5d8c5a15da6eab965c48
Nov 03 17:57:09.655034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.654633    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.655034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.654708    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.655034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.654736    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:09.700886 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.700816    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4" podStartSLOduration=2.7007949 podStartE2EDuration="2.7007949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:09.700509664 +0000 UTC m=+1409.159680652" watchObservedRunningTime="2026-11-03 17:57:09.7007949 +0000 UTC m=+1409.159965887"
Nov 03 17:57:09.762589 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.762521    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:09.791645 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.791543    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:09.896164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.896069    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podStartSLOduration=1.896046254 podStartE2EDuration="1.896046254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:09.892944659 +0000 UTC m=+1409.352115646" watchObservedRunningTime="2026-11-03 17:57:09.896046254 +0000 UTC m=+1409.355217237"
Nov 03 17:57:09.911638 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.911589    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b425fc-14ba-440e-a596-310a9261dec8-kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/50b425fc-14ba-440e-a596-310a9261dec8-kube-api-access-xq9bg\") pod \"50b425fc-14ba-440e-a596-310a9261dec8\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") "
Nov 03 17:57:09.913430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.913387    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-openshift-ca" pod "50b425fc-14ba-440e-a596-310a9261dec8" (UID: "50b425fc-14ba-440e-a596-310a9261dec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:09.913590 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.912855    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-openshift-ca\") pod \"50b425fc-14ba-440e-a596-310a9261dec8\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") "
Nov 03 17:57:09.913826 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.913810    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-xks-ca\") pod \"50b425fc-14ba-440e-a596-310a9261dec8\" (UID: \"50b425fc-14ba-440e-a596-310a9261dec8\") "
Nov 03 17:57:09.915088 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.915069    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:09.918091 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.918073    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:09.918357 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.918331    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b425fc-14ba-440e-a596-310a9261dec8-kube-api-access-xq9bg" pod "50b425fc-14ba-440e-a596-310a9261dec8" (UID: "50b425fc-14ba-440e-a596-310a9261dec8"). InnerVolumeSpecName "kube-api-access-xq9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:09.919974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:09.919947    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-xks-ca" pod "50b425fc-14ba-440e-a596-310a9261dec8" (UID: "50b425fc-14ba-440e-a596-310a9261dec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:10.016505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.016471    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b425fc-14ba-440e-a596-310a9261dec8-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:10.016721 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.016706    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9bg\" (UniqueName: \"kubernetes.io/projected/50b425fc-14ba-440e-a596-310a9261dec8-kube-api-access-xq9bg\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:10.027718 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.027596    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podStartSLOduration=2.027575061 podStartE2EDuration="2.027575061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:10.0213373 +0000 UTC m=+1409.480508287" watchObservedRunningTime="2026-11-03 17:57:10.027575061 +0000 UTC m=+1409.486746048"
Nov 03 17:57:10.697039 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.696993    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" event={"ID":"8cf94c4a-b570-4670-942c-8e981619c397","Type":"ContainerStarted","Data":"33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2"}
Nov 03 17:57:10.712655 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.711438    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" event={"ID":"fc030dce-3454-4c6c-8ebb-e532e89676ef","Type":"ContainerStarted","Data":"d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944"}
Nov 03 17:57:10.712997 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.712967    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" event={"ID":"fc030dce-3454-4c6c-8ebb-e532e89676ef","Type":"ContainerStarted","Data":"3ee7eb17d3a63daee094c49925f0fc8e74cb1df212ef5d8c5a15da6eab965c48"}
Nov 03 17:57:10.811301 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.811255    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:10.811583 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.811539    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:10.840932 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.819583    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" event={"ID":"75c84f63-1f56-4d39-af09-ee4a3b8f35e1","Type":"ContainerStarted","Data":"a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69"}
Nov 03 17:57:10.840932 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.819699    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" event={"ID":"75c84f63-1f56-4d39-af09-ee4a3b8f35e1","Type":"ContainerStarted","Data":"5cbcaf843b46d8cbfb3f7cf08063b861e68b130fa15b2b6088f5ef7cae0abb3b"}
Nov 03 17:57:10.851767 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.850941    4493 generic.go:356] "Generic (PLEG): container finished" podID="511d605d-a149-4ab6-a6a2-07c267a474b3" containerID="7a78b5baffec4a3195f0842aacb91b6d6a9ceedb236ef17b4871dbf1067d9212" exitCode=0
Nov 03 17:57:10.851767 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.851067    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4" event={"ID":"511d605d-a149-4ab6-a6a2-07c267a474b3","Type":"ContainerDied","Data":"7a78b5baffec4a3195f0842aacb91b6d6a9ceedb236ef17b4871dbf1067d9212"}
Nov 03 17:57:10.859150 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.859007    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm" event={"ID":"50b425fc-14ba-440e-a596-310a9261dec8","Type":"ContainerDied","Data":"d7b40f654d093beadbdffbc4ad23f0d079f438d082ec83a455342d8e6f732948"}
Nov 03 17:57:10.859150 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.859085    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7b40f654d093beadbdffbc4ad23f0d079f438d082ec83a455342d8e6f732948"
Nov 03 17:57:10.868955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.868930    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"
Nov 03 17:57:10.935900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.934983    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:10.935900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.935046    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:10.974899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:10.967832    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podStartSLOduration=2.9678085 podStartE2EDuration="2.9678085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:10.870891609 +0000 UTC m=+1410.330062606" watchObservedRunningTime="2026-11-03 17:57:10.9678085 +0000 UTC m=+1410.426979487"
Nov 03 17:57:11.052906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.052484    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podStartSLOduration=3.052459097 podStartE2EDuration="3.052459097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:11.026212624 +0000 UTC m=+1410.485383612" watchObservedRunningTime="2026-11-03 17:57:11.052459097 +0000 UTC m=+1410.511630085"
Nov 03 17:57:11.131077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.130935    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:11.138151 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.133449    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.138151 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.133507    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:11.198916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.181496    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:11.198916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.181636    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.198916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.181678    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:11.250722 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.250693    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:11.272721 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.272690    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.281651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.281613    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:11.724626 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.724583    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:11.767435 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.767396    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.767731 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.767688    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:11.826752 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.824952    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 17:57:11.851178 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.850750    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:11.921362 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.921229    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:11.921362 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.921300    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:11.925326 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.924974    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:11.978957 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.977966    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:11.978957 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:11.978339    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:57:12.094353 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.093102    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.126247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.125295    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.126247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.126140    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.126247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.126179    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:12.187918 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.187370    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.187918 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.187431    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:12.204182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.203305    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:12.244955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.231315    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.244955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.231381    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:12.275899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.273962    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.334103 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.329959    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.481965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.378688    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.481965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.419775    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.514596 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.501803    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.791721 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.731328    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:12.791721 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.732492    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:57:12.821823 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.817647    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:12.821823 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.817703    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:12.829763 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.828948    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.033001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.890288    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.033001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.953270    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.033001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.958235    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.033001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.958283    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:13.033001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.981968    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.033001 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:12.995051    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 17:57:13.102673 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.101557    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.108902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.103816    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podStartSLOduration=5.103798691 podStartE2EDuration="5.103798691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:11.325795503 +0000 UTC m=+1410.784966492" watchObservedRunningTime="2026-11-03 17:57:13.103798691 +0000 UTC m=+1412.562969678"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.178799    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.202924    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.202989    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.203027    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.203063    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.203094    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.203127    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.203159    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.204749    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.206681    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.233706    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4" event={"ID":"511d605d-a149-4ab6-a6a2-07c267a474b3","Type":"ContainerDied","Data":"a7de60bb3d5d231364445cf164b605b365db61939b29a513785e2987e0ea287e"}
Nov 03 17:57:13.236906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.233763    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7de60bb3d5d231364445cf164b605b365db61939b29a513785e2987e0ea287e"
Nov 03 17:57:13.237688 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.236999    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.237688 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.237050    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:13.286901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.256248    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:13.286901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.257589    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.286901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.257629    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:13.286901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.276072    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.286901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.276484    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:57:13.333901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.333416    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.333901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.333472    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:13.334140 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.334005    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.354042    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.389267    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.389992    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.431030    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.451021    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.504981    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.510999    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.506890    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-xks-ca\") pod \"511d605d-a149-4ab6-a6a2-07c267a474b3\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") "
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.520511    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-openshift-ca\") pod \"511d605d-a149-4ab6-a6a2-07c267a474b3\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") "
Nov 03 17:57:13.526987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.520585    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511d605d-a149-4ab6-a6a2-07c267a474b3-kube-api-access-njllm\" (UniqueName: \"kubernetes.io/projected/511d605d-a149-4ab6-a6a2-07c267a474b3-kube-api-access-njllm\") pod \"511d605d-a149-4ab6-a6a2-07c267a474b3\" (UID: \"511d605d-a149-4ab6-a6a2-07c267a474b3\") "
Nov 03 17:57:13.632150 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.632108    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-openshift-ca" pod "511d605d-a149-4ab6-a6a2-07c267a474b3" (UID: "511d605d-a149-4ab6-a6a2-07c267a474b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:13.643106 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.643070    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511d605d-a149-4ab6-a6a2-07c267a474b3-kube-api-access-njllm" pod "511d605d-a149-4ab6-a6a2-07c267a474b3" (UID: "511d605d-a149-4ab6-a6a2-07c267a474b3"). InnerVolumeSpecName "kube-api-access-njllm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:13.685951 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.653128    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:13.785955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.762912    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.785955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.762968    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:13.785955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.763039    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.840901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.787590    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njllm\" (UniqueName: \"kubernetes.io/projected/511d605d-a149-4ab6-a6a2-07c267a474b3-kube-api-access-njllm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:13.843858 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.841473    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.910901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.908252    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-xks-ca" pod "511d605d-a149-4ab6-a6a2-07c267a474b3" (UID: "511d605d-a149-4ab6-a6a2-07c267a474b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:13.910901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.909734    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:13.941899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.939589    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Nov 03 17:57:13.941899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.940076    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:13.941899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:13.940114    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:14.048005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.026899    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/511d605d-a149-4ab6-a6a2-07c267a474b3-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:14.048005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.028435    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.048005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.028744    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.114899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.049445    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.190596 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.115671    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.191118 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.116403    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.191296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.191272    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:14.191400 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.190546    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.191523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.191504    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:14.191615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.181573    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.204975 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.204934    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.214250 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.214218    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.214454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.214432    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:14.307380 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.307286    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 17:57:14.307683 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.307587    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.478236 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.478189    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.507836 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.507787    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.624449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.588001    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:57:14.624449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.596539    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.764145 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.764107    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:14.768610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.768579    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:14.796597 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.767503    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.857466 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.857420    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:14.969556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.969508    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.240980 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:14.971270    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"
Nov 03 17:57:15.241140 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.114316    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.241344 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.137797    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.241595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.150328    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.241687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.241626    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:15.241687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.206195    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.241913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.150529    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.262013 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.151583    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.262273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.262245    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:15.262373 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.206238    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.262749 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.206256    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.263104 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.232366    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.263430 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.182746    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.263589 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.263566    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:15.263693 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.232314    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.263787 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.263772    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:15.291899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.265619    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.321493 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.311025    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.484843 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.484001    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.508659 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.508563    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.636937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.611021    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.636937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.611331    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:57:15.636937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.614258    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.695929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.652972    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.825636 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.825528    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.863066 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.862963    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:15.939051 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.939013    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.939298 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.939270    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:15.939512 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.939493    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:15.939639 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:15.939617    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:16.279899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.276494    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.279899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.276554    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:16.280278 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.280252    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.280414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.280393    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:16.280808 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.280787    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.281758 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281735    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:16.296005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281596    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:16.298077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281615    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 17:57:16.298077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281632    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:57:16.298077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281650    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:16.298077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281666    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 17:57:16.298077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281683    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:16.298077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.281568    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.331024 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.330980    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.491603 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.491564    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.520012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.519965    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.574116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.574019    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.631447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.631394    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.631930 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.631708    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.780034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.777377    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:16.780034 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.777450    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:16.858949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.856074    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:16.974901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:16.971240    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.021136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.021100    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.021392 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.021353    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:17.153955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.151561    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.153955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.151793    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:17.178901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.170293    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.178901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.170349    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:17.214493 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.214022    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.240084 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.239974    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.243899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.242111    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:17.322621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.296659    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.322621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.296842    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.322621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.296954    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.322621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.298045    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.322621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.318277    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.322621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.319527    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.322621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.320570    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:17.352282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.343301    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.500901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.493157    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.522436 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.520403    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.640901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.640004    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.644899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.641326    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.656899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.654616    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.767114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.767026    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.767355 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.767329    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:17.860031 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.859984    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:17.934121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.934084    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:17.934396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.934369    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:17.986954 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:17.982014    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.059075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.058969    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.117772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.117733    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"]
Nov 03 17:57:18.119561 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.119534    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.119790 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.119765    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:18.129894 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.124006    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0506f-pn7jb"]
Nov 03 17:57:18.224612 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.224576    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.346676 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.342698    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.364038 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.344337    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.364499 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.344376    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:18.390271 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.344414    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.390916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.231497    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.391129 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.391087    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:18.391267 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.344495    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:18.413896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.391495    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:18.413896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.344472    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 17:57:18.413896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.344438    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.493383 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.493323    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.557900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.536657    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.586414 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.584974    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.657124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.657013    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.657721 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.657691    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:57:18.768612 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.768576    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.769004 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.768975    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 17:57:18.818228 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.818196    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8306dce6-39a4-4f03-bd6c-bdf46e84a8dc" path="/var/lib/kubelet/pods/8306dce6-39a4-4f03-bd6c-bdf46e84a8dc/volumes"
Nov 03 17:57:18.864897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.862957    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:18.875201 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.875174    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:57:18.902891 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.902440    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:57:18.903173 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.903156    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:57:18.929379 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.929285    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 17:57:18.969520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.969467    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 17:57:18.986071 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:18.985968    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 17:57:19.243273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.243237    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.243909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.243849    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:19.369836 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.369799    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.370139 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.370106    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 03 17:57:19.371358 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.371338    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:57:19.375275 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.372569    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:57:19.376991 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.376961    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:19.382215 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.382189    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:19.419685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.382410    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 17:57:19.427982 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.427949    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 17:57:19.470556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.377718    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:57:19.470556 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.470124    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:19.478937 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.478901    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 17:57:19.576808 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.576705    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:19.727225 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.727182    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 17:57:19.768642 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.768607    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:57:19.771457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.771429    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:57:19.799232 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.799192    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:19.867955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.864375    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:57:19.867955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:19.866777    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:57:20.004312 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.004283    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:57:20.075057 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.075029    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:57:20.082164 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.082141    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:57:20.138297 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.138211    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:57:20.258010 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.257978    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 17:57:20.258572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.258541    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 17:57:20.283791 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.283753    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:20.285334 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.285314    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:20.288456 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.288436    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 17:57:20.478712 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.478684    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:20.489590 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.489545    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:20.492735 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.492682    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 17:57:20.655161 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.655130    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:20.656611 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.656566    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:20.658270 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.658087    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 17:57:20.926461 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.926392    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:20.927056 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.927024    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:20.928743 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:20.928720    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 17:57:21.217982 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:21.217951    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:21.218240 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:21.218222    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:21.219075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:21.219053    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 17:57:24.155983 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.155940    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"]
Nov 03 17:57:24.156480 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.156467    4493 cpu_manager.go:414] "containerMap: removing container" podUID="50b425fc-14ba-440e-a596-310a9261dec8" containerName="revoke-keys"
Nov 03 17:57:24.156537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.156481    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b425fc-14ba-440e-a596-310a9261dec8" containerName="revoke-keys"
Nov 03 17:57:24.156537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.156507    4493 cpu_manager.go:414] "containerMap: removing container" podUID="511d605d-a149-4ab6-a6a2-07c267a474b3" containerName="revoke-keys"
Nov 03 17:57:24.156537 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.156513    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511d605d-a149-4ab6-a6a2-07c267a474b3" containerName="revoke-keys"
Nov 03 17:57:24.156631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.156594    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="511d605d-a149-4ab6-a6a2-07c267a474b3" containerName="revoke-keys"
Nov 03 17:57:24.156631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.156607    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b425fc-14ba-440e-a596-310a9261dec8" containerName="revoke-keys"
Nov 03 17:57:24.161423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.161403    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.217340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.217309    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:57:24.217572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.217449    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:57:24.217572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.217492    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:57:24.217734 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.217510    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:57:24.219890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.219848    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"]
Nov 03 17:57:24.303427 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.303393    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.303644 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.303624    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.303690 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.303654    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5987\" (UniqueName: \"kubernetes.io/projected/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-kube-api-access-d5987\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.404999 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.404958    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.404999 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.405000    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5987\" (UniqueName: \"kubernetes.io/projected/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-kube-api-access-d5987\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.405255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.405072    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.405656 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.405628    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.407669 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.407619    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.469153 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.469123    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5987\" (UniqueName: \"kubernetes.io/projected/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-kube-api-access-d5987\") pod \"maas-api-revoke-sub-7218fd6b-44ldt\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.474815 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.474794    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:24.880106 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:24.880076    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c475a0_21f7_4c6b_a1dc_43ba64dd7270.slice/crio-8c74f06eccfaff1ec230b75a4034ef9bf8e06b28c786764a4cb177f296ec218f WatchSource:0}: Error finding container 8c74f06eccfaff1ec230b75a4034ef9bf8e06b28c786764a4cb177f296ec218f: Status 404 returned error can't find the container with id 8c74f06eccfaff1ec230b75a4034ef9bf8e06b28c786764a4cb177f296ec218f
Nov 03 17:57:24.880106 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:24.880078    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"]
Nov 03 17:57:25.050570 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.050538    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt" event={"ID":"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270","Type":"ContainerStarted","Data":"8c74f06eccfaff1ec230b75a4034ef9bf8e06b28c786764a4cb177f296ec218f"}
Nov 03 17:57:25.354226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.354196    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"]
Nov 03 17:57:25.358369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.358348    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.417993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.417922    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"]
Nov 03 17:57:25.522059 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.522028    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.522197 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.522070    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5vtq\" (UniqueName: \"kubernetes.io/projected/a69c8f25-5592-4c7a-9156-521993aa5d54-kube-api-access-v5vtq\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.522197 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.522181    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.623101 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.623069    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.623265 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.623109    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5vtq\" (UniqueName: \"kubernetes.io/projected/a69c8f25-5592-4c7a-9156-521993aa5d54-kube-api-access-v5vtq\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.623265 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.623178    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.623805 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.623779    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.625656 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.625632    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.687113 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.687040    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5vtq\" (UniqueName: \"kubernetes.io/projected/a69c8f25-5592-4c7a-9156-521993aa5d54-kube-api-access-v5vtq\") pod \"maas-api-revoke-sub-499cc594-5559n\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:25.973399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:25.973356    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:26.071648 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:26.071620    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"]
Nov 03 17:57:26.072831 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:26.072804    4493 generic.go:356] "Generic (PLEG): container finished" podID="c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" containerID="43609cab1051d412e76db2a53548ddec132ef23f27ef9201a77118e754ea14de" exitCode=0
Nov 03 17:57:26.072978 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:26.072900    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt" event={"ID":"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270","Type":"ContainerDied","Data":"43609cab1051d412e76db2a53548ddec132ef23f27ef9201a77118e754ea14de"}
Nov 03 17:57:26.075208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:26.075181    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a35cf42d-2z5mc"]
Nov 03 17:57:26.356549 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:26.356522    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69c8f25_5592_4c7a_9156_521993aa5d54.slice/crio-ecb77f6254befd742540e124eff8a26c3f18e5339f561e4acd54990e23e81d84 WatchSource:0}: Error finding container ecb77f6254befd742540e124eff8a26c3f18e5339f561e4acd54990e23e81d84: Status 404 returned error can't find the container with id ecb77f6254befd742540e124eff8a26c3f18e5339f561e4acd54990e23e81d84
Nov 03 17:57:26.357180 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:26.357157    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"]
Nov 03 17:57:26.714189 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:26.714156    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4224491-4ba8-40d6-894f-b5c381cae384" path="/var/lib/kubelet/pods/a4224491-4ba8-40d6-894f-b5c381cae384/volumes"
Nov 03 17:57:27.101511 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.092198    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n" event={"ID":"a69c8f25-5592-4c7a-9156-521993aa5d54","Type":"ContainerStarted","Data":"c3707ca42148ff62aa8a444b01fe5ee966aaff556823f091aac5466fad40beac"}
Nov 03 17:57:27.101511 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.092256    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n" event={"ID":"a69c8f25-5592-4c7a-9156-521993aa5d54","Type":"ContainerStarted","Data":"ecb77f6254befd742540e124eff8a26c3f18e5339f561e4acd54990e23e81d84"}
Nov 03 17:57:27.740804 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.740477    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:27.812438 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.812397    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-kube-api-access-d5987\" (UniqueName: \"kubernetes.io/projected/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-kube-api-access-d5987\") pod \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") "
Nov 03 17:57:27.812739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.812716    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-openshift-ca\") pod \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") "
Nov 03 17:57:27.812964 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.812945    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-xks-ca\") pod \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\" (UID: \"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270\") "
Nov 03 17:57:27.849939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.836575    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-xks-ca" pod "c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" (UID: "c0c475a0-21f7-4c6b-a1dc-43ba64dd7270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:27.850791 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.850757    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-openshift-ca" pod "c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" (UID: "c0c475a0-21f7-4c6b-a1dc-43ba64dd7270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:27.887934 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.886409    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-kube-api-access-d5987" pod "c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" (UID: "c0c475a0-21f7-4c6b-a1dc-43ba64dd7270"). InnerVolumeSpecName "kube-api-access-d5987". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:27.943521 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.899420    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n" podStartSLOduration=2.899397119 podStartE2EDuration="2.899397119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:27.500522239 +0000 UTC m=+1426.959693248" watchObservedRunningTime="2026-11-03 17:57:27.899397119 +0000 UTC m=+1427.358568107"
Nov 03 17:57:27.943521 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.935111    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5987\" (UniqueName: \"kubernetes.io/projected/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-kube-api-access-d5987\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:27.943521 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.935143    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:27.943521 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:27.935159    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:28.169396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.169298    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"
Nov 03 17:57:28.220902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.193969    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt" event={"ID":"c0c475a0-21f7-4c6b-a1dc-43ba64dd7270","Type":"ContainerDied","Data":"8c74f06eccfaff1ec230b75a4034ef9bf8e06b28c786764a4cb177f296ec218f"}
Nov 03 17:57:28.220902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.194080    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c74f06eccfaff1ec230b75a4034ef9bf8e06b28c786764a4cb177f296ec218f"
Nov 03 17:57:28.225431 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.225391    4493 generic.go:356] "Generic (PLEG): container finished" podID="a69c8f25-5592-4c7a-9156-521993aa5d54" containerID="c3707ca42148ff62aa8a444b01fe5ee966aaff556823f091aac5466fad40beac" exitCode=0
Nov 03 17:57:28.229433 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.229319    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n" event={"ID":"a69c8f25-5592-4c7a-9156-521993aa5d54","Type":"ContainerDied","Data":"c3707ca42148ff62aa8a444b01fe5ee966aaff556823f091aac5466fad40beac"}
Nov 03 17:57:28.440707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.436963    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:28.440707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.437269    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 17:57:28.452576 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.451968    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:28.452576 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.452247    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:28.452576 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.452409    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:28.565061 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.565017    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:28.616728 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.616686    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:28.620234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.617236    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:28.830855 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.830805    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:28.879947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:28.858013    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.041903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.040634    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.082955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.081975    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.203899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.202977    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.439988    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.440354    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.440663    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.440724    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.458517    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.458854    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.459135    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.459420    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.459479    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:57:29.489002 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.459517    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:57:29.525003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.516968    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.576066 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.575982    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 17:57:29.729290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.588605    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:57:29.729290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.642249    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.729290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.643365    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.799955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.759006    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:57:29.799955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.759084    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:57:29.799955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.788011    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:29.837008 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.836961    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.020753 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:29.859831    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.028899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.021057    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:57:30.038924 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.032992    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:57:30.093951 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.045847    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.125916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.108282    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:57:30.155913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.128533    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.155913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.130499    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:57:30.221921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.221060    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.221921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.221508    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:57:30.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.454051    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.462607    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 17:57:30.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.463421    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.470963    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:30.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.471280    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 17:57:30.553898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.552958    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.553898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.553393    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:57:30.595905 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.595072    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 17:57:30.667945 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.666054    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.667945 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.666412    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:57:30.920573 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.888043    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:30.920573 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.888106    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:30.920573 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.893238    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:57:30.920573 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.893281    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.035899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:30.984762    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.035899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.009499    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.035899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.030414    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:57:31.061946 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.052987    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.113900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.105283    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:31.131737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.128358    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:57:31.131737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.130465    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 17:57:31.212523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.159369    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:57:31.212523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.188681    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:57:31.229374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.225069    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.256333 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.254309    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.445756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.430165    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-openshift-ca\") pod \"a69c8f25-5592-4c7a-9156-521993aa5d54\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") "
Nov 03 17:57:31.445756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.430480    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-xks-ca\") pod \"a69c8f25-5592-4c7a-9156-521993aa5d54\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") "
Nov 03 17:57:31.445756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.430524    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69c8f25-5592-4c7a-9156-521993aa5d54-kube-api-access-v5vtq\" (UniqueName: \"kubernetes.io/projected/a69c8f25-5592-4c7a-9156-521993aa5d54-kube-api-access-v5vtq\") pod \"a69c8f25-5592-4c7a-9156-521993aa5d54\" (UID: \"a69c8f25-5592-4c7a-9156-521993aa5d54\") "
Nov 03 17:57:31.445756 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.435565    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:57:31.485260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.479971    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.485260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.480449    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:31.485793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.485747    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-openshift-ca" pod "a69c8f25-5592-4c7a-9156-521993aa5d54" (UID: "a69c8f25-5592-4c7a-9156-521993aa5d54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:31.680114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.569222    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:31.680114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.573460    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69c8f25-5592-4c7a-9156-521993aa5d54-kube-api-access-v5vtq" pod "a69c8f25-5592-4c7a-9156-521993aa5d54" (UID: "a69c8f25-5592-4c7a-9156-521993aa5d54"). InnerVolumeSpecName "kube-api-access-v5vtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:31.680114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.611522    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.724998 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.720157    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:31.724998 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.720273    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:57:31.848505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.751139    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5vtq\" (UniqueName: \"kubernetes.io/projected/a69c8f25-5592-4c7a-9156-521993aa5d54-kube-api-access-v5vtq\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:31.848505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.797547    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:57:31.848505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.797618    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:57:31.848505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.837065    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"
Nov 03 17:57:31.848505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.838389    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n" event={"ID":"a69c8f25-5592-4c7a-9156-521993aa5d54","Type":"ContainerDied","Data":"ecb77f6254befd742540e124eff8a26c3f18e5339f561e4acd54990e23e81d84"}
Nov 03 17:57:31.848505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.838436    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb77f6254befd742540e124eff8a26c3f18e5339f561e4acd54990e23e81d84"
Nov 03 17:57:31.910913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.907740    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-xks-ca" pod "a69c8f25-5592-4c7a-9156-521993aa5d54" (UID: "a69c8f25-5592-4c7a-9156-521993aa5d54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:31.922396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.914293    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:57:31.923041 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.922903    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:57:31.968919 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:31.966741    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69c8f25-5592-4c7a-9156-521993aa5d54-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:32.005966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:32.005935    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:57:32.014923 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:32.014847    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:57:32.028390 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:32.028364    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:57:32.029906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:32.029854    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:57:32.031679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:32.031659    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:57:35.710322 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.710290    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"]
Nov 03 17:57:35.710781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.710768    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" containerName="revoke-keys"
Nov 03 17:57:35.710781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.710779    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" containerName="revoke-keys"
Nov 03 17:57:35.710914 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.710804    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a69c8f25-5592-4c7a-9156-521993aa5d54" containerName="revoke-keys"
Nov 03 17:57:35.710914 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.710809    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69c8f25-5592-4c7a-9156-521993aa5d54" containerName="revoke-keys"
Nov 03 17:57:35.710914 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.710914    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69c8f25-5592-4c7a-9156-521993aa5d54" containerName="revoke-keys"
Nov 03 17:57:35.711044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.710928    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" containerName="revoke-keys"
Nov 03 17:57:35.729237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.729204    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.774782 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.774746    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"]
Nov 03 17:57:35.819966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.819933    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:57:35.819966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.819933    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:57:35.820161 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.820032    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:57:35.827732 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.827694    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.827950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.827926    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47658\" (UniqueName: \"kubernetes.io/projected/1912ba5f-fa32-4379-b2fd-654ba9e2c004-kube-api-access-47658\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.828112 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.828093    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.834355 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.834334    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:57:35.929006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.928978    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.929193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.929075    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47658\" (UniqueName: \"kubernetes.io/projected/1912ba5f-fa32-4379-b2fd-654ba9e2c004-kube-api-access-47658\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.929316 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.929294    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.929633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.929607    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.931808 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.931781    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:35.993353 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:35.993322    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47658\" (UniqueName: \"kubernetes.io/projected/1912ba5f-fa32-4379-b2fd-654ba9e2c004-kube-api-access-47658\") pod \"maas-api-revoke-sub-f3b69506-wbrq8\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:36.044834 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.044805    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:36.128331 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.128302    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"]
Nov 03 17:57:36.137174 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.136465    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.184993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.184954    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"]
Nov 03 17:57:36.233583 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.233554    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.233761 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.233591    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxsn\" (UniqueName: \"kubernetes.io/projected/03427e23-9290-4ff8-9f8e-188849cc105b-kube-api-access-2lxsn\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.233861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.233841    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.335503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.335418    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.335503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.335469    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxsn\" (UniqueName: \"kubernetes.io/projected/03427e23-9290-4ff8-9f8e-188849cc105b-kube-api-access-2lxsn\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.335709 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.335671    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.336102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.336082    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.338298 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.338276    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.399299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.399269    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxsn\" (UniqueName: \"kubernetes.io/projected/03427e23-9290-4ff8-9f8e-188849cc105b-kube-api-access-2lxsn\") pod \"maas-api-revoke-sub-d02c14c3-fm7vd\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.435696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.435667    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"]
Nov 03 17:57:36.436257 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:36.436232    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1912ba5f_fa32_4379_b2fd_654ba9e2c004.slice/crio-86da49b66b3f66d8c9d94126661b4ba9fc2dba7cef90772702b71c4a953135c2 WatchSource:0}: Error finding container 86da49b66b3f66d8c9d94126661b4ba9fc2dba7cef90772702b71c4a953135c2: Status 404 returned error can't find the container with id 86da49b66b3f66d8c9d94126661b4ba9fc2dba7cef90772702b71c4a953135c2
Nov 03 17:57:36.459457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.459437    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:36.840087 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.840056    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"]
Nov 03 17:57:36.840520 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:36.840174    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03427e23_9290_4ff8_9f8e_188849cc105b.slice/crio-3557f029be8eb524758f48383030dd9790645ed6177bec6c7927cfa56b8db63c WatchSource:0}: Error finding container 3557f029be8eb524758f48383030dd9790645ed6177bec6c7927cfa56b8db63c: Status 404 returned error can't find the container with id 3557f029be8eb524758f48383030dd9790645ed6177bec6c7927cfa56b8db63c
Nov 03 17:57:36.876958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.876922    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8" event={"ID":"1912ba5f-fa32-4379-b2fd-654ba9e2c004","Type":"ContainerStarted","Data":"3da44fe28b5b1e58ef545a0aed763c2384bb26f454bfca5661910e3bbd4532a3"}
Nov 03 17:57:36.877091 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.876994    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8" event={"ID":"1912ba5f-fa32-4379-b2fd-654ba9e2c004","Type":"ContainerStarted","Data":"86da49b66b3f66d8c9d94126661b4ba9fc2dba7cef90772702b71c4a953135c2"}
Nov 03 17:57:36.878303 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.878274    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd" event={"ID":"03427e23-9290-4ff8-9f8e-188849cc105b","Type":"ContainerStarted","Data":"3557f029be8eb524758f48383030dd9790645ed6177bec6c7927cfa56b8db63c"}
Nov 03 17:57:36.993339 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:57:36.993307    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:57:36.997477 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:36.997433    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8" podStartSLOduration=1.997420532 podStartE2EDuration="1.997420532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:36.997210833 +0000 UTC m=+1436.456381820" watchObservedRunningTime="2026-11-03 17:57:36.997420532 +0000 UTC m=+1436.456591517"
Nov 03 17:57:37.571179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.571108    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"]
Nov 03 17:57:37.575169 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.575153    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.634918 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.634885    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"]
Nov 03 17:57:37.754846 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.754810    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvvh\" (UniqueName: \"kubernetes.io/projected/2e108ba7-433c-45b5-a659-7aaddd0b3993-kube-api-access-mxvvh\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.755039 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.754960    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.755039 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.755011    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.855967 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.855865    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.856397 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.856074    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.856552 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.856523    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvvh\" (UniqueName: \"kubernetes.io/projected/2e108ba7-433c-45b5-a659-7aaddd0b3993-kube-api-access-mxvvh\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.856673 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.856606    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.858633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.858611    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:37.886670 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.886636    4493 generic.go:356] "Generic (PLEG): container finished" podID="03427e23-9290-4ff8-9f8e-188849cc105b" containerID="35c701d9992865d3c33d1c6b9247ac3a3f8458b43135f15f57c373de117c5290" exitCode=0
Nov 03 17:57:37.886812 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.886675    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd" event={"ID":"03427e23-9290-4ff8-9f8e-188849cc105b","Type":"ContainerDied","Data":"35c701d9992865d3c33d1c6b9247ac3a3f8458b43135f15f57c373de117c5290"}
Nov 03 17:57:37.888329 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.888305    4493 generic.go:356] "Generic (PLEG): container finished" podID="1912ba5f-fa32-4379-b2fd-654ba9e2c004" containerID="3da44fe28b5b1e58ef545a0aed763c2384bb26f454bfca5661910e3bbd4532a3" exitCode=0
Nov 03 17:57:37.888422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.888380    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8" event={"ID":"1912ba5f-fa32-4379-b2fd-654ba9e2c004","Type":"ContainerDied","Data":"3da44fe28b5b1e58ef545a0aed763c2384bb26f454bfca5661910e3bbd4532a3"}
Nov 03 17:57:37.920651 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:37.920620    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvvh\" (UniqueName: \"kubernetes.io/projected/2e108ba7-433c-45b5-a659-7aaddd0b3993-kube-api-access-mxvvh\") pod \"maas-api-revoke-sub-b98b5253-lwnjg\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:38.188224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.188143    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:38.569520 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.569489    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"]
Nov 03 17:57:38.569942 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:38.569917    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e108ba7_433c_45b5_a659_7aaddd0b3993.slice/crio-03c0caaa2e31d094f0cfb6a39666076eea2b7f946971c6cc84c26f03797b2b75 WatchSource:0}: Error finding container 03c0caaa2e31d094f0cfb6a39666076eea2b7f946971c6cc84c26f03797b2b75: Status 404 returned error can't find the container with id 03c0caaa2e31d094f0cfb6a39666076eea2b7f946971c6cc84c26f03797b2b75
Nov 03 17:57:38.806089 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.806062    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"]
Nov 03 17:57:38.811269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.811249    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.880794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.871303    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.880794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.871553    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.880794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.871586    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhk8\" (UniqueName: \"kubernetes.io/projected/37110d7b-d6ce-43c8-9584-2de563362f4c-kube-api-access-5vhk8\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.880794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.876704    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"]
Nov 03 17:57:38.900319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.900276    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg" event={"ID":"2e108ba7-433c-45b5-a659-7aaddd0b3993","Type":"ContainerStarted","Data":"03c0caaa2e31d094f0cfb6a39666076eea2b7f946971c6cc84c26f03797b2b75"}
Nov 03 17:57:38.980255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.974687    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.980255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.974742    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhk8\" (UniqueName: \"kubernetes.io/projected/37110d7b-d6ce-43c8-9584-2de563362f4c-kube-api-access-5vhk8\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.980255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.974851    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.980255 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.975680    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:38.981552 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:38.981503    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:39.035374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.035346    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:39.039824 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.039800    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhk8\" (UniqueName: \"kubernetes.io/projected/37110d7b-d6ce-43c8-9584-2de563362f4c-kube-api-access-5vhk8\") pod \"maas-api-revoke-sub-4540545a-znp4n\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:39.076134 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.076062    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-xks-ca\") pod \"03427e23-9290-4ff8-9f8e-188849cc105b\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") "
Nov 03 17:57:39.076260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.076141    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-openshift-ca\") pod \"03427e23-9290-4ff8-9f8e-188849cc105b\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") "
Nov 03 17:57:39.076319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.076307    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03427e23-9290-4ff8-9f8e-188849cc105b-kube-api-access-2lxsn\" (UniqueName: \"kubernetes.io/projected/03427e23-9290-4ff8-9f8e-188849cc105b-kube-api-access-2lxsn\") pod \"03427e23-9290-4ff8-9f8e-188849cc105b\" (UID: \"03427e23-9290-4ff8-9f8e-188849cc105b\") "
Nov 03 17:57:39.076515 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.076488    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-openshift-ca" pod "03427e23-9290-4ff8-9f8e-188849cc105b" (UID: "03427e23-9290-4ff8-9f8e-188849cc105b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:39.076926 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.076848    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:39.078482 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.078454    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-xks-ca" pod "03427e23-9290-4ff8-9f8e-188849cc105b" (UID: "03427e23-9290-4ff8-9f8e-188849cc105b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:39.078580 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.078484    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03427e23-9290-4ff8-9f8e-188849cc105b-kube-api-access-2lxsn" pod "03427e23-9290-4ff8-9f8e-188849cc105b" (UID: "03427e23-9290-4ff8-9f8e-188849cc105b"). InnerVolumeSpecName "kube-api-access-2lxsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:39.089567 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.089544    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:39.133095 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.133062    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:39.178288 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.178252    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1912ba5f-fa32-4379-b2fd-654ba9e2c004-kube-api-access-47658\" (UniqueName: \"kubernetes.io/projected/1912ba5f-fa32-4379-b2fd-654ba9e2c004-kube-api-access-47658\") pod \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") "
Nov 03 17:57:39.178445 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.178365    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-xks-ca\") pod \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") "
Nov 03 17:57:39.178445 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.178433    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-openshift-ca\") pod \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\" (UID: \"1912ba5f-fa32-4379-b2fd-654ba9e2c004\") "
Nov 03 17:57:39.178927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.178890    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lxsn\" (UniqueName: \"kubernetes.io/projected/03427e23-9290-4ff8-9f8e-188849cc105b-kube-api-access-2lxsn\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:39.178927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.178918    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03427e23-9290-4ff8-9f8e-188849cc105b-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:39.179265 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.179244    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-openshift-ca" pod "1912ba5f-fa32-4379-b2fd-654ba9e2c004" (UID: "1912ba5f-fa32-4379-b2fd-654ba9e2c004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:39.181239 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.181213    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-xks-ca" pod "1912ba5f-fa32-4379-b2fd-654ba9e2c004" (UID: "1912ba5f-fa32-4379-b2fd-654ba9e2c004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:39.181315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.181254    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1912ba5f-fa32-4379-b2fd-654ba9e2c004-kube-api-access-47658" pod "1912ba5f-fa32-4379-b2fd-654ba9e2c004" (UID: "1912ba5f-fa32-4379-b2fd-654ba9e2c004"). InnerVolumeSpecName "kube-api-access-47658". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:39.280479 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.280442    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47658\" (UniqueName: \"kubernetes.io/projected/1912ba5f-fa32-4379-b2fd-654ba9e2c004-kube-api-access-47658\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:39.280479 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.280475    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:39.280479 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.280487    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1912ba5f-fa32-4379-b2fd-654ba9e2c004-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:39.530665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.530632    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"]
Nov 03 17:57:39.530779 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:39.530684    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37110d7b_d6ce_43c8_9584_2de563362f4c.slice/crio-ba9920e83d669707f88fa0809153e78eb1c3933d8427ee0293d676f6e37a00ef WatchSource:0}: Error finding container ba9920e83d669707f88fa0809153e78eb1c3933d8427ee0293d676f6e37a00ef: Status 404 returned error can't find the container with id ba9920e83d669707f88fa0809153e78eb1c3933d8427ee0293d676f6e37a00ef
Nov 03 17:57:39.909305 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.909272    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd" event={"ID":"03427e23-9290-4ff8-9f8e-188849cc105b","Type":"ContainerDied","Data":"3557f029be8eb524758f48383030dd9790645ed6177bec6c7927cfa56b8db63c"}
Nov 03 17:57:39.909305 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.909296    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"
Nov 03 17:57:39.909814 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.909328    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3557f029be8eb524758f48383030dd9790645ed6177bec6c7927cfa56b8db63c"
Nov 03 17:57:39.911281 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.911240    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n" event={"ID":"37110d7b-d6ce-43c8-9584-2de563362f4c","Type":"ContainerStarted","Data":"171354be469ecf1264df6003b45112acd21eb91b625eeda938dfa5f90e1bf3ab"}
Nov 03 17:57:39.911446 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.911288    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n" event={"ID":"37110d7b-d6ce-43c8-9584-2de563362f4c","Type":"ContainerStarted","Data":"ba9920e83d669707f88fa0809153e78eb1c3933d8427ee0293d676f6e37a00ef"}
Nov 03 17:57:39.912820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.912800    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"
Nov 03 17:57:39.912972 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.912839    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8" event={"ID":"1912ba5f-fa32-4379-b2fd-654ba9e2c004","Type":"ContainerDied","Data":"86da49b66b3f66d8c9d94126661b4ba9fc2dba7cef90772702b71c4a953135c2"}
Nov 03 17:57:39.912972 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.912901    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86da49b66b3f66d8c9d94126661b4ba9fc2dba7cef90772702b71c4a953135c2"
Nov 03 17:57:39.914783 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.914760    4493 generic.go:356] "Generic (PLEG): container finished" podID="2e108ba7-433c-45b5-a659-7aaddd0b3993" containerID="d76362e9c85352cf3b14e6d543acc42069f18295f5dcf92e95cdf963ea395433" exitCode=0
Nov 03 17:57:39.914910 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:39.914835    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg" event={"ID":"2e108ba7-433c-45b5-a659-7aaddd0b3993","Type":"ContainerDied","Data":"d76362e9c85352cf3b14e6d543acc42069f18295f5dcf92e95cdf963ea395433"}
Nov 03 17:57:40.037108 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.037062    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n" podStartSLOduration=2.037049698 podStartE2EDuration="2.037049698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:40.03442977 +0000 UTC m=+1439.493600756" watchObservedRunningTime="2026-11-03 17:57:40.037049698 +0000 UTC m=+1439.496220723"
Nov 03 17:57:40.086003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.085920    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"]
Nov 03 17:57:40.086572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.086553    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1912ba5f-fa32-4379-b2fd-654ba9e2c004" containerName="revoke-keys"
Nov 03 17:57:40.086687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.086575    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1912ba5f-fa32-4379-b2fd-654ba9e2c004" containerName="revoke-keys"
Nov 03 17:57:40.086687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.086615    4493 cpu_manager.go:414] "containerMap: removing container" podUID="03427e23-9290-4ff8-9f8e-188849cc105b" containerName="revoke-keys"
Nov 03 17:57:40.086687 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.086623    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03427e23-9290-4ff8-9f8e-188849cc105b" containerName="revoke-keys"
Nov 03 17:57:40.086839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.086782    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="03427e23-9290-4ff8-9f8e-188849cc105b" containerName="revoke-keys"
Nov 03 17:57:40.086839 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.086803    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1912ba5f-fa32-4379-b2fd-654ba9e2c004" containerName="revoke-keys"
Nov 03 17:57:40.090352 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.090330    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.145598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.145574    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-73a606-openshift-default-dockercfg-6xzts\""
Nov 03 17:57:40.161048 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.161013    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"]
Nov 03 17:57:40.191505 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.191472    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84ab85b-542a-40f5-9712-4516c3799a06-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.191662 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.191629    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrn2t\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-kube-api-access-wrn2t\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.191741 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.191721    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.191804 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.191783    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.191912 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.191892    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.191951 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.191933    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.192003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.191975    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.192159 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.192133    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.192227 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.192212    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84ab85b-542a-40f5-9712-4516c3799a06-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294078 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294043    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrn2t\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-kube-api-access-wrn2t\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294121    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294164    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294220 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294191    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294372 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294218    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294372 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294255    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294372 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294306    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294480 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294371    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84ab85b-542a-40f5-9712-4516c3799a06-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294532 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294491    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84ab85b-542a-40f5-9712-4516c3799a06-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294652 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294628    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294729 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294654    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294729 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294705    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.294827 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.294812    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.295234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.295164    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84ab85b-542a-40f5-9712-4516c3799a06-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.296765 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.296744    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.296917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.296900    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84ab85b-542a-40f5-9712-4516c3799a06-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.357813 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.357739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.357968 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.357945    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrn2t\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-kube-api-access-wrn2t\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.405950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.405924    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:40.852386 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.852356    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"]
Nov 03 17:57:40.855926 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:40.855893    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84ab85b_542a_40f5_9712_4516c3799a06.slice/crio-54088df8d51b968c02581302e1b213b251d1e27e08ef0332a4ca3dfd472ffc8c WatchSource:0}: Error finding container 54088df8d51b968c02581302e1b213b251d1e27e08ef0332a4ca3dfd472ffc8c: Status 404 returned error can't find the container with id 54088df8d51b968c02581302e1b213b251d1e27e08ef0332a4ca3dfd472ffc8c
Nov 03 17:57:40.858575 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.858534    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:40.858698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.858627    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:40.858698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.858673    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:40.923345 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.923323    4493 generic.go:356] "Generic (PLEG): container finished" podID="37110d7b-d6ce-43c8-9584-2de563362f4c" containerID="171354be469ecf1264df6003b45112acd21eb91b625eeda938dfa5f90e1bf3ab" exitCode=0
Nov 03 17:57:40.923702 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.923390    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n" event={"ID":"37110d7b-d6ce-43c8-9584-2de563362f4c","Type":"ContainerDied","Data":"171354be469ecf1264df6003b45112acd21eb91b625eeda938dfa5f90e1bf3ab"}
Nov 03 17:57:40.924418 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.924401    4493 scope.go:122] "RemoveContainer" containerID="4d5562f7cee78b02ca86ee11166a44b87167acd393b3485b5f3879c84aad192f"
Nov 03 17:57:40.925690 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.925667    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" event={"ID":"e84ab85b-542a-40f5-9712-4516c3799a06","Type":"ContainerStarted","Data":"54088df8d51b968c02581302e1b213b251d1e27e08ef0332a4ca3dfd472ffc8c"}
Nov 03 17:57:40.936412 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.936386    4493 scope.go:122] "RemoveContainer" containerID="bda2b32d935d3286a56f412be64587133e135d737b9888dcd1c6dd97be4942c0"
Nov 03 17:57:40.949116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.949089    4493 scope.go:122] "RemoveContainer" containerID="3e5febf232357c7f4849040426cdcb67b52b0e87c7f11a204713c64790faeac9"
Nov 03 17:57:40.960998 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.960973    4493 scope.go:122] "RemoveContainer" containerID="fa24c3ded1b6bc193afabd74498ad517cf1799b4976c002182ad05e875f32ca2"
Nov 03 17:57:40.974250 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.974225    4493 scope.go:122] "RemoveContainer" containerID="66b0c996036b0f73520af48ac7640a51e3b4d87fedc2b7e403c966802fcb168b"
Nov 03 17:57:40.990378 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:40.990357    4493 scope.go:122] "RemoveContainer" containerID="2ecddc094b8d7131115948a027fb5fb4ce6ff40d29021ad27292ba6034cd7ea8"
Nov 03 17:57:41.005818 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.003819    4493 scope.go:122] "RemoveContainer" containerID="064c6bf230ac563b7ea50972e4ae67e6e8a048c88fbf8a4616f19195cc31e3e1"
Nov 03 17:57:41.016156 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.015851    4493 scope.go:122] "RemoveContainer" containerID="089e1422e0aea13b026d9d0ff844bf3ffa6b7c12f893ab65b8809ed48e414fa5"
Nov 03 17:57:41.030861 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.030839    4493 scope.go:122] "RemoveContainer" containerID="72088f6b72d5d27b38cf8538268d233f274510edd0eb30d306ac0b03bd003387"
Nov 03 17:57:41.089855 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.089754    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:41.105653 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.105629    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-openshift-ca\") pod \"2e108ba7-433c-45b5-a659-7aaddd0b3993\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") "
Nov 03 17:57:41.105830 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.105767    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e108ba7-433c-45b5-a659-7aaddd0b3993-kube-api-access-mxvvh\" (UniqueName: \"kubernetes.io/projected/2e108ba7-433c-45b5-a659-7aaddd0b3993-kube-api-access-mxvvh\") pod \"2e108ba7-433c-45b5-a659-7aaddd0b3993\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") "
Nov 03 17:57:41.105899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.105847    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-xks-ca\") pod \"2e108ba7-433c-45b5-a659-7aaddd0b3993\" (UID: \"2e108ba7-433c-45b5-a659-7aaddd0b3993\") "
Nov 03 17:57:41.106043 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.106022    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-openshift-ca" pod "2e108ba7-433c-45b5-a659-7aaddd0b3993" (UID: "2e108ba7-433c-45b5-a659-7aaddd0b3993"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:41.106501 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.106478    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:41.108120 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.108090    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-xks-ca" pod "2e108ba7-433c-45b5-a659-7aaddd0b3993" (UID: "2e108ba7-433c-45b5-a659-7aaddd0b3993"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:41.108310 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.108291    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e108ba7-433c-45b5-a659-7aaddd0b3993-kube-api-access-mxvvh" pod "2e108ba7-433c-45b5-a659-7aaddd0b3993" (UID: "2e108ba7-433c-45b5-a659-7aaddd0b3993"). InnerVolumeSpecName "kube-api-access-mxvvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:41.207402 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.207366    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxvvh\" (UniqueName: \"kubernetes.io/projected/2e108ba7-433c-45b5-a659-7aaddd0b3993-kube-api-access-mxvvh\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:41.207402 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.207400    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e108ba7-433c-45b5-a659-7aaddd0b3993-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:41.947341 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.947300    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" event={"ID":"e84ab85b-542a-40f5-9712-4516c3799a06","Type":"ContainerStarted","Data":"2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230"}
Nov 03 17:57:41.948960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.948936    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"
Nov 03 17:57:41.949075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.948958    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg" event={"ID":"2e108ba7-433c-45b5-a659-7aaddd0b3993","Type":"ContainerDied","Data":"03c0caaa2e31d094f0cfb6a39666076eea2b7f946971c6cc84c26f03797b2b75"}
Nov 03 17:57:41.949075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:41.949002    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c0caaa2e31d094f0cfb6a39666076eea2b7f946971c6cc84c26f03797b2b75"
Nov 03 17:57:42.144460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.144393    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podStartSLOduration=2.144373379 podStartE2EDuration="2.144373379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:42.141809485 +0000 UTC m=+1441.600980496" watchObservedRunningTime="2026-11-03 17:57:42.144373379 +0000 UTC m=+1441.603544366"
Nov 03 17:57:42.155126 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.155092    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:42.249771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.249721    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37110d7b-d6ce-43c8-9584-2de563362f4c-kube-api-access-5vhk8\" (UniqueName: \"kubernetes.io/projected/37110d7b-d6ce-43c8-9584-2de563362f4c-kube-api-access-5vhk8\") pod \"37110d7b-d6ce-43c8-9584-2de563362f4c\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") "
Nov 03 17:57:42.249771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.249766    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-openshift-ca\") pod \"37110d7b-d6ce-43c8-9584-2de563362f4c\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") "
Nov 03 17:57:42.250099 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.249798    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-xks-ca\") pod \"37110d7b-d6ce-43c8-9584-2de563362f4c\" (UID: \"37110d7b-d6ce-43c8-9584-2de563362f4c\") "
Nov 03 17:57:42.250194 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.250164    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-openshift-ca" pod "37110d7b-d6ce-43c8-9584-2de563362f4c" (UID: "37110d7b-d6ce-43c8-9584-2de563362f4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:42.250593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.250565    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:42.252126 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.252104    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-xks-ca" pod "37110d7b-d6ce-43c8-9584-2de563362f4c" (UID: "37110d7b-d6ce-43c8-9584-2de563362f4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:42.252244 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.252220    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37110d7b-d6ce-43c8-9584-2de563362f4c-kube-api-access-5vhk8" pod "37110d7b-d6ce-43c8-9584-2de563362f4c" (UID: "37110d7b-d6ce-43c8-9584-2de563362f4c"). InnerVolumeSpecName "kube-api-access-5vhk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:42.351921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.351882    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vhk8\" (UniqueName: \"kubernetes.io/projected/37110d7b-d6ce-43c8-9584-2de563362f4c-kube-api-access-5vhk8\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:42.351921 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.351918    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37110d7b-d6ce-43c8-9584-2de563362f4c-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:42.406827 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.406791    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:42.408170 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.408141    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 17:57:42.408317 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.408198    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 17:57:42.957532 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.957503    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"
Nov 03 17:57:42.957933 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.957544    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n" event={"ID":"37110d7b-d6ce-43c8-9584-2de563362f4c","Type":"ContainerDied","Data":"ba9920e83d669707f88fa0809153e78eb1c3933d8427ee0293d676f6e37a00ef"}
Nov 03 17:57:42.957933 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:42.957597    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9920e83d669707f88fa0809153e78eb1c3933d8427ee0293d676f6e37a00ef"
Nov 03 17:57:43.406575 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:43.406544    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 17:57:43.406752 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:43.406614    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 17:57:44.407029 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:44.406998    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 17:57:44.407424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:44.407048    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 17:57:45.406451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:45.406408    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 17:57:45.406659 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:45.406472    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 17:57:46.407266 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:46.407229    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 17:57:46.407643 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:46.407290    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 17:57:47.408248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:47.406790    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Nov 03 17:57:47.408248 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:47.406846    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Nov 03 17:57:48.411478 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:48.411443    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:48.411960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:48.411898    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:48.412832 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:48.412811    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 17:57:50.865758 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.865725    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"]
Nov 03 17:57:50.866377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.866361    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2e108ba7-433c-45b5-a659-7aaddd0b3993" containerName="revoke-keys"
Nov 03 17:57:50.866424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.866380    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e108ba7-433c-45b5-a659-7aaddd0b3993" containerName="revoke-keys"
Nov 03 17:57:50.866424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.866390    4493 cpu_manager.go:414] "containerMap: removing container" podUID="37110d7b-d6ce-43c8-9584-2de563362f4c" containerName="revoke-keys"
Nov 03 17:57:50.866424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.866394    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37110d7b-d6ce-43c8-9584-2de563362f4c" containerName="revoke-keys"
Nov 03 17:57:50.866526 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.866516    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e108ba7-433c-45b5-a659-7aaddd0b3993" containerName="revoke-keys"
Nov 03 17:57:50.866563 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.866527    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="37110d7b-d6ce-43c8-9584-2de563362f4c" containerName="revoke-keys"
Nov 03 17:57:50.871335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.871318    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:50.929613 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.929580    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"]
Nov 03 17:57:50.967215 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.967180    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:50.967448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.967418    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:50.967560 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.967470    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rnb\" (UniqueName: \"kubernetes.io/projected/a3749470-e5ad-49df-bd13-ed8215373bf7-kube-api-access-64rnb\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:50.974578 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.974555    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:57:50.974691 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.974561    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:57:50.974691 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.974602    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:57:50.975522 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:50.975506    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:57:51.068268 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.068239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:51.068462 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.068274    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rnb\" (UniqueName: \"kubernetes.io/projected/a3749470-e5ad-49df-bd13-ed8215373bf7-kube-api-access-64rnb\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:51.068462 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.068383    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:51.069075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.069055    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:51.070737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.070715    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:51.132585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.132520    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rnb\" (UniqueName: \"kubernetes.io/projected/a3749470-e5ad-49df-bd13-ed8215373bf7-kube-api-access-64rnb\") pod \"maas-api-revoke-sub-671c255a-jqxr4\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:51.184940 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.184911    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:51.578714 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:51.578686    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"]
Nov 03 17:57:51.578994 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:57:51.578967    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3749470_e5ad_49df_bd13_ed8215373bf7.slice/crio-fa89e80dc236237e7c07b14945a57090439c53388983fcb6ae3aec7021613b73 WatchSource:0}: Error finding container fa89e80dc236237e7c07b14945a57090439c53388983fcb6ae3aec7021613b73: Status 404 returned error can't find the container with id fa89e80dc236237e7c07b14945a57090439c53388983fcb6ae3aec7021613b73
Nov 03 17:57:52.025773 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.025735    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4" event={"ID":"a3749470-e5ad-49df-bd13-ed8215373bf7","Type":"ContainerStarted","Data":"16c4cc301fe9e1cdabcb411c546257ae02c8a9bf64393c8fb696650c6887011c"}
Nov 03 17:57:52.026310 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.025799    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4" event={"ID":"a3749470-e5ad-49df-bd13-ed8215373bf7","Type":"ContainerStarted","Data":"fa89e80dc236237e7c07b14945a57090439c53388983fcb6ae3aec7021613b73"}
Nov 03 17:57:52.102397 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.102305    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"]
Nov 03 17:57:52.112235 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.112204    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.150377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.148226    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4" podStartSLOduration=2.148206966 podStartE2EDuration="2.148206966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:52.146346506 +0000 UTC m=+1451.605517519" watchObservedRunningTime="2026-11-03 17:57:52.148206966 +0000 UTC m=+1451.607377956"
Nov 03 17:57:52.299898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.284930    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"]
Nov 03 17:57:52.325897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.313744    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b879l\" (UniqueName: \"kubernetes.io/projected/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-kube-api-access-b879l\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.325897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.313978    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.325897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.314049    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.428138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.417764    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.428138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.417859    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.428138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.417981    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b879l\" (UniqueName: \"kubernetes.io/projected/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-kube-api-access-b879l\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.428138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.419219    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.428138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.426239    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.497899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.495399    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b879l\" (UniqueName: \"kubernetes.io/projected/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-kube-api-access-b879l\") pod \"maas-api-revoke-sub-0750fdee-9zq4p\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:52.759897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:52.754222    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:53.071737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:53.071652    4493 generic.go:356] "Generic (PLEG): container finished" podID="a3749470-e5ad-49df-bd13-ed8215373bf7" containerID="16c4cc301fe9e1cdabcb411c546257ae02c8a9bf64393c8fb696650c6887011c" exitCode=0
Nov 03 17:57:53.072478 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:53.072449    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4" event={"ID":"a3749470-e5ad-49df-bd13-ed8215373bf7","Type":"ContainerDied","Data":"16c4cc301fe9e1cdabcb411c546257ae02c8a9bf64393c8fb696650c6887011c"}
Nov 03 17:57:53.641953 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:53.630244    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:53.677965 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:53.668972    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:53.746309 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:53.746041    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:53.775906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:53.774006    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:53.966599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:53.942032    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Nov 03 17:57:54.032114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.009753    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"]
Nov 03 17:57:54.117292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.093097    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.166915 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.158331    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.166915 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.166614    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.347615 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.345604    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:54.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.474206    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.479228 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.479188    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.576609 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.547998    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.587132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.553455    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p" event={"ID":"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a","Type":"ContainerStarted","Data":"3f7646e7f41cdeec12a3271b367ee8dc77f893753c18c706a8f686c58c342e4d"}
Nov 03 17:57:54.587424 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.553958    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.863939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.836701    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:54.974987 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:54.959186    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:57:55.141133 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.141090    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.167912 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.142257    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.167912 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.142591    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:57:55.168456 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.168420    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:57:55.193920 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.181368    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:57:55.212032 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.211987    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.219899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.212629    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:57:55.325689 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.325646    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.367716 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.350219    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"]
Nov 03 17:57:55.367716 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.356861    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.367716 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.358987    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Nov 03 17:57:55.367716 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.359288    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:57:55.367716 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.359661    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:57:55.411917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.392984    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.492787    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.492901    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.493928    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494012    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494055    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqx2d\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-kube-api-access-lqx2d\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494107    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494177    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494214    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494317    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494372    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.497931 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.494394    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.499402 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.499370    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:57:55.610023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.599028    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:55.610023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.599432    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 17:57:55.610023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.599751    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:57:55.610023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.599814    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:57:55.649902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.624374    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.650999 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.636282    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.651308 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.651283    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.677327 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.677296    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqx2d\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-kube-api-access-lqx2d\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.677605 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.677589    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.677774 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.677762    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.677913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.677897    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.678135 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.678117    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.678291 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.678274    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.678402 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.678387    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.682897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.636912    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.709899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.673938    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.723017 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.722978    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.764907 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.725223    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.801900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.697080    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.802165 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.755085    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.802296 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.758177    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.805941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.784580    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.830966 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.787230    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"]
Nov 03 17:57:55.831237 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.812463    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.880950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.840508    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:55.899916 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.896559    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqx2d\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-kube-api-access-lqx2d\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:55.990495 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:55.990404    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:56.095900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.070583    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:56.149289 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.149186    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.150345 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.150323    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:57:56.183935 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.175040    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.234954 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.212943    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.271934 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.263592    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:56.327174 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.327120    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.327865 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.327843    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:57:56.360085 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.359687    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.522697 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.516588    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.522697 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.520564    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:57:56.576239 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.572481    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:56.610453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.606081    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 17:57:56.610453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.606454    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:56.760161 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.747291    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 17:57:56.920696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.763077    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:57:56.920696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.893162    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:57:56.920696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:56.763515    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.183644 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.183558    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.184205 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.184179    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:57:57.184347 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.184331    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded"
Nov 03 17:57:57.216899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.208892    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:57:57.216899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.209545    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:57:57.216899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.213941    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:57.216899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.214279    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.337899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.330213    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.386896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.338590    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4" event={"ID":"a3749470-e5ad-49df-bd13-ed8215373bf7","Type":"ContainerDied","Data":"fa89e80dc236237e7c07b14945a57090439c53388983fcb6ae3aec7021613b73"}
Nov 03 17:57:57.387282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.367013    4493 generic.go:356] "Generic (PLEG): container finished" podID="1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" containerID="1f3c926c39eb91705048ababd90f1f37ad9138495cca608bbb8f649f24dd1ce7" exitCode=0
Nov 03 17:57:57.412999 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.412966    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa89e80dc236237e7c07b14945a57090439c53388983fcb6ae3aec7021613b73"
Nov 03 17:57:57.413258 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.413233    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p" event={"ID":"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a","Type":"ContainerDied","Data":"1f3c926c39eb91705048ababd90f1f37ad9138495cca608bbb8f649f24dd1ce7"}
Nov 03 17:57:57.576050 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.565519    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 17:57:57.751006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.750960    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.783830    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3749470-e5ad-49df-bd13-ed8215373bf7-kube-api-access-64rnb\" (UniqueName: \"kubernetes.io/projected/a3749470-e5ad-49df-bd13-ed8215373bf7-kube-api-access-64rnb\") pod \"a3749470-e5ad-49df-bd13-ed8215373bf7\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") "
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.783922    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-xks-ca\") pod \"a3749470-e5ad-49df-bd13-ed8215373bf7\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") "
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.783977    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-openshift-ca\") pod \"a3749470-e5ad-49df-bd13-ed8215373bf7\" (UID: \"a3749470-e5ad-49df-bd13-ed8215373bf7\") "
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.759637    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.784771    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.760827    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.819085    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-openshift-ca" pod "a3749470-e5ad-49df-bd13-ed8215373bf7" (UID: "a3749470-e5ad-49df-bd13-ed8215373bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.820185    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.862584    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.862741    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:57.874952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.862865    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:57:57.982560 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:57.978938    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.216548 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.216506    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:57:58.225913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.218379    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-xks-ca" pod "a3749470-e5ad-49df-bd13-ed8215373bf7" (UID: "a3749470-e5ad-49df-bd13-ed8215373bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:58.273329 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.247406    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.273329 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.272470    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3749470-e5ad-49df-bd13-ed8215373bf7-kube-api-access-64rnb" pod "a3749470-e5ad-49df-bd13-ed8215373bf7" (UID: "a3749470-e5ad-49df-bd13-ed8215373bf7"). InnerVolumeSpecName "kube-api-access-64rnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:58.274338 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.274298    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded"
Nov 03 17:57:58.275041 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.274499    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 17:57:58.293472 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.293404    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.339369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.339324    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.345941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.345847    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rnb\" (UniqueName: \"kubernetes.io/projected/a3749470-e5ad-49df-bd13-ed8215373bf7-kube-api-access-64rnb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:58.352817 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.352790    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3749470-e5ad-49df-bd13-ed8215373bf7-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:58.371806 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.356280    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"]
Nov 03 17:57:58.378955 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.378925    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:57:58.496964 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.496934    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:57:58.576434 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.576394    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:57:58.596976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.596948    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:57:58.597219 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.597207    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:57:58.602792 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.602764    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:57:58.603094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.603078    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:57:58.608392 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.607343    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:57:58.630498 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.630469    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:57:58.633704 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.633676    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" event={"ID":"8b257bb2-2d65-4ada-a050-c5260dcc4dea","Type":"ContainerStarted","Data":"c5131ce47a8dd951efee81405f0b97856ea4decd35abf1f2447bb91566a4a75a"}
Nov 03 17:57:58.634450 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.634430    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"
Nov 03 17:57:58.832134 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.832102    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:57:58.901083 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.901049    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-xks-ca\") pod \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") "
Nov 03 17:57:58.901269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.901181    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-kube-api-access-b879l\" (UniqueName: \"kubernetes.io/projected/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-kube-api-access-b879l\") pod \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") "
Nov 03 17:57:58.901269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.901211    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-openshift-ca\") pod \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\" (UID: \"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a\") "
Nov 03 17:57:58.901864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.901808    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-openshift-ca" pod "1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" (UID: "1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:57:58.902003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.901957    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:58.904159 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.904135    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-xks-ca" pod "1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" (UID: "1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:57:58.904245 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:58.904164    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-kube-api-access-b879l" pod "1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" (UID: "1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a"). InnerVolumeSpecName "kube-api-access-b879l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:57:59.003659 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.003627    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:59.003659 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.003658    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b879l\" (UniqueName: \"kubernetes.io/projected/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a-kube-api-access-b879l\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:57:59.180619 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.180510    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podStartSLOduration=4.18049647 podStartE2EDuration="4.18049647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:59.059210568 +0000 UTC m=+1458.518381556" watchObservedRunningTime="2026-11-03 17:57:59.18049647 +0000 UTC m=+1458.639667456"
Nov 03 17:57:59.286157 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.286107    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:57:59.287567 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.287486    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 17:57:59.287567 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.287535    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 17:57:59.656691 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.656630    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" event={"ID":"8b257bb2-2d65-4ada-a050-c5260dcc4dea","Type":"ContainerStarted","Data":"097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977"}
Nov 03 17:57:59.659369 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.659307    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p" event={"ID":"1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a","Type":"ContainerDied","Data":"3f7646e7f41cdeec12a3271b367ee8dc77f893753c18c706a8f686c58c342e4d"}
Nov 03 17:57:59.659502 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.659389    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f7646e7f41cdeec12a3271b367ee8dc77f893753c18c706a8f686c58c342e4d"
Nov 03 17:57:59.659502 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:57:59.659463    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"
Nov 03 17:58:00.286444 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:00.286411    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 17:58:00.286833 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:00.286464    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 17:58:01.287142 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.287100    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 17:58:01.287540 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.287165    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 17:58:01.654231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.654147    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"]
Nov 03 17:58:01.655182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.655156    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3749470-e5ad-49df-bd13-ed8215373bf7" containerName="revoke-keys"
Nov 03 17:58:01.655182 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.655182    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3749470-e5ad-49df-bd13-ed8215373bf7" containerName="revoke-keys"
Nov 03 17:58:01.655378 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.655232    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" containerName="revoke-keys"
Nov 03 17:58:01.655378 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.655240    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" containerName="revoke-keys"
Nov 03 17:58:01.655484 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.655391    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3749470-e5ad-49df-bd13-ed8215373bf7" containerName="revoke-keys"
Nov 03 17:58:01.655484 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.655414    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" containerName="revoke-keys"
Nov 03 17:58:01.659211 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.659191    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.714583 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.714549    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:58:01.765486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.761710    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"]
Nov 03 17:58:01.766947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.766253    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:58:01.766947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.766377    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:58:01.766947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.766489    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:58:01.773855 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.773429    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.773855 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.773487    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.773855 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.773551    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hm9w\" (UniqueName: \"kubernetes.io/projected/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-kube-api-access-5hm9w\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.875179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.875142    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.875179 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.875182    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hm9w\" (UniqueName: \"kubernetes.io/projected/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-kube-api-access-5hm9w\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.875397 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.875386    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.876052 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.876028    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.878010 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.877988    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.946583 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.946506    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hm9w\" (UniqueName: \"kubernetes.io/projected/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-kube-api-access-5hm9w\") pod \"maas-api-revoke-sub-2d520831-qczh7\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:01.977284 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:01.977251    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:02.287290 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:02.287260    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 17:58:02.287656 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:02.287321    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 17:58:02.394489 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:02.394463    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"]
Nov 03 17:58:02.394900 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:02.394847    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ed1d0d_85a5_41f3_ae17_a6ce22efc00a.slice/crio-23cc87846d1be8905350710dfe40850e3d2cadcfc2e0e14cce48ec65ffb31778 WatchSource:0}: Error finding container 23cc87846d1be8905350710dfe40850e3d2cadcfc2e0e14cce48ec65ffb31778: Status 404 returned error can't find the container with id 23cc87846d1be8905350710dfe40850e3d2cadcfc2e0e14cce48ec65ffb31778
Nov 03 17:58:02.694129 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:02.694036    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7" event={"ID":"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a","Type":"ContainerStarted","Data":"23cc87846d1be8905350710dfe40850e3d2cadcfc2e0e14cce48ec65ffb31778"}
Nov 03 17:58:03.157583 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.156375    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"]
Nov 03 17:58:03.164914 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.161244    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.202382 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.202349    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.202736 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.202710    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.203012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.202988    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kd6\" (UniqueName: \"kubernetes.io/projected/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-kube-api-access-s5kd6\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.287958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.287376    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 17:58:03.287958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.287467    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 17:58:03.291736 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.288909    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"]
Nov 03 17:58:03.305486 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.305432    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kd6\" (UniqueName: \"kubernetes.io/projected/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-kube-api-access-s5kd6\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.305689 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.305670    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.305797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.305759    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.307250 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.307184    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.379064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.378286    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.382063 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.382022    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kd6\" (UniqueName: \"kubernetes.io/projected/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-kube-api-access-s5kd6\") pod \"maas-api-revoke-sub-4edab22e-gjj4n\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.486475 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.486441    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:03.703994 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.703912    4493 generic.go:356] "Generic (PLEG): container finished" podID="14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" containerID="d852c519f0a27f237f60cbd934f49975dd2664575fb1a7f4bc6fac63b040a03a" exitCode=0
Nov 03 17:58:03.704137 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.704060    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7" event={"ID":"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a","Type":"ContainerDied","Data":"d852c519f0a27f237f60cbd934f49975dd2664575fb1a7f4bc6fac63b040a03a"}
Nov 03 17:58:03.947938 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:03.947907    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"]
Nov 03 17:58:03.948319 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:03.948296    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890b3ab1_3e60_4bad_8ec1_e19ad007d7c0.slice/crio-88bf148c958c18b05e9d4c25ffffd409e09967018a5395d257b60df4e73bca50 WatchSource:0}: Error finding container 88bf148c958c18b05e9d4c25ffffd409e09967018a5395d257b60df4e73bca50: Status 404 returned error can't find the container with id 88bf148c958c18b05e9d4c25ffffd409e09967018a5395d257b60df4e73bca50
Nov 03 17:58:04.287392 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.287347    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 17:58:04.287568 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.287411    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 17:58:04.771754 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.771720    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n" event={"ID":"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0","Type":"ContainerStarted","Data":"29bf9e2e47850fb9c6d910ab836f6efc15ba244ce0f07439f6ebb84b5d8123f8"}
Nov 03 17:58:04.772201 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.771783    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n" event={"ID":"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0","Type":"ContainerStarted","Data":"88bf148c958c18b05e9d4c25ffffd409e09967018a5395d257b60df4e73bca50"}
Nov 03 17:58:04.886884 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.886830    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"]
Nov 03 17:58:04.891645 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.891606    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:04.901684 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.901633    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n" podStartSLOduration=1.901618677 podStartE2EDuration="1.901618677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:04.898390917 +0000 UTC m=+1464.357561902" watchObservedRunningTime="2026-11-03 17:58:04.901618677 +0000 UTC m=+1464.360789664"
Nov 03 17:58:04.950949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.950915    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:04.951400 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.951380    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:04.951650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.951635    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzrw\" (UniqueName: \"kubernetes.io/projected/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-kube-api-access-bvzrw\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:04.952232 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.952201    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"]
Nov 03 17:58:04.978231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:04.978203    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:05.053134 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.053030    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-xks-ca\") pod \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") "
Nov 03 17:58:05.053134 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.053125    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-openshift-ca\") pod \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") "
Nov 03 17:58:05.053377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.053250    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-kube-api-access-5hm9w\" (UniqueName: \"kubernetes.io/projected/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-kube-api-access-5hm9w\") pod \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\" (UID: \"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a\") "
Nov 03 17:58:05.053459 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.053434    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-openshift-ca" pod "14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" (UID: "14ed1d0d-85a5-41f3-ae17-a6ce22efc00a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:05.053601 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.053579    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzrw\" (UniqueName: \"kubernetes.io/projected/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-kube-api-access-bvzrw\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:05.053693 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.053680    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:05.054633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.053886    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:05.054633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.054041    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:05.054633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.054583    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:05.055633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.055599    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-xks-ca" pod "14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" (UID: "14ed1d0d-85a5-41f3-ae17-a6ce22efc00a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:05.056132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.056114    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:05.056219 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.056197    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-kube-api-access-5hm9w" pod "14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" (UID: "14ed1d0d-85a5-41f3-ae17-a6ce22efc00a"). InnerVolumeSpecName "kube-api-access-5hm9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:05.119180 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.119148    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzrw\" (UniqueName: \"kubernetes.io/projected/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-kube-api-access-bvzrw\") pod \"maas-api-revoke-sub-c274391f-qt67h\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:05.156136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.156084    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:05.156136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.156138    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hm9w\" (UniqueName: \"kubernetes.io/projected/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a-kube-api-access-5hm9w\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:05.215500 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.215466    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:05.293969 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.287482    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 17:58:05.293969 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.287540    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 17:58:05.608334 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:05.608299    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f720ee_ecfe_4e21_a2c3_ffe0519d901a.slice/crio-cecdb3ceebce130ab980de9a7633573837ea9ab7456a966fcf965a4acfefe8f7 WatchSource:0}: Error finding container cecdb3ceebce130ab980de9a7633573837ea9ab7456a966fcf965a4acfefe8f7: Status 404 returned error can't find the container with id cecdb3ceebce130ab980de9a7633573837ea9ab7456a966fcf965a4acfefe8f7
Nov 03 17:58:05.609127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.608926    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"]
Nov 03 17:58:05.779060 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.779022    4493 generic.go:356] "Generic (PLEG): container finished" podID="890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" containerID="29bf9e2e47850fb9c6d910ab836f6efc15ba244ce0f07439f6ebb84b5d8123f8" exitCode=0
Nov 03 17:58:05.779470 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.779119    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n" event={"ID":"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0","Type":"ContainerDied","Data":"29bf9e2e47850fb9c6d910ab836f6efc15ba244ce0f07439f6ebb84b5d8123f8"}
Nov 03 17:58:05.780686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.780659    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h" event={"ID":"10f720ee-ecfe-4e21-a2c3-ffe0519d901a","Type":"ContainerStarted","Data":"cecdb3ceebce130ab980de9a7633573837ea9ab7456a966fcf965a4acfefe8f7"}
Nov 03 17:58:05.782231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.782210    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"
Nov 03 17:58:05.782231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.782219    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7" event={"ID":"14ed1d0d-85a5-41f3-ae17-a6ce22efc00a","Type":"ContainerDied","Data":"23cc87846d1be8905350710dfe40850e3d2cadcfc2e0e14cce48ec65ffb31778"}
Nov 03 17:58:05.782386 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:05.782257    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23cc87846d1be8905350710dfe40850e3d2cadcfc2e0e14cce48ec65ffb31778"
Nov 03 17:58:06.309453 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.309425    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:58:06.309793 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.309767    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:58:06.310725 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.310702    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 17:58:06.791571 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.791542    4493 generic.go:356] "Generic (PLEG): container finished" podID="10f720ee-ecfe-4e21-a2c3-ffe0519d901a" containerID="4d79b04c866ea31a28017c4206d8cd19380456392cb7081d1214876cc1a2cd01" exitCode=0
Nov 03 17:58:06.792004 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.791631    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h" event={"ID":"10f720ee-ecfe-4e21-a2c3-ffe0519d901a","Type":"ContainerDied","Data":"4d79b04c866ea31a28017c4206d8cd19380456392cb7081d1214876cc1a2cd01"}
Nov 03 17:58:06.936526 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.936470    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:06.981205 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.981176    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-xks-ca\") pod \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") "
Nov 03 17:58:06.981357 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.981268    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-openshift-ca\") pod \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") "
Nov 03 17:58:06.981472 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.981457    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-kube-api-access-s5kd6\" (UniqueName: \"kubernetes.io/projected/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-kube-api-access-s5kd6\") pod \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\" (UID: \"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0\") "
Nov 03 17:58:06.981722 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.981691    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-openshift-ca" pod "890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" (UID: "890b3ab1-3e60-4bad-8ec1-e19ad007d7c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:06.982044 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.981955    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:06.983327 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.983302    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-xks-ca" pod "890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" (UID: "890b3ab1-3e60-4bad-8ec1-e19ad007d7c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:06.983560 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:06.983538    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-kube-api-access-s5kd6" pod "890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" (UID: "890b3ab1-3e60-4bad-8ec1-e19ad007d7c0"). InnerVolumeSpecName "kube-api-access-s5kd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:07.018502 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:58:06.994241    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:58:07.083964 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:07.083927    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:07.083964 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:07.083962    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5kd6\" (UniqueName: \"kubernetes.io/projected/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0-kube-api-access-s5kd6\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:07.800853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:07.800823    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"
Nov 03 17:58:07.800853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:07.800844    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n" event={"ID":"890b3ab1-3e60-4bad-8ec1-e19ad007d7c0","Type":"ContainerDied","Data":"88bf148c958c18b05e9d4c25ffffd409e09967018a5395d257b60df4e73bca50"}
Nov 03 17:58:07.801307 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:07.800913    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88bf148c958c18b05e9d4c25ffffd409e09967018a5395d257b60df4e73bca50"
Nov 03 17:58:07.960946 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:07.960916    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:08.097772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.097689    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-kube-api-access-bvzrw\" (UniqueName: \"kubernetes.io/projected/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-kube-api-access-bvzrw\") pod \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") "
Nov 03 17:58:08.097772 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.097764    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-openshift-ca\") pod \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") "
Nov 03 17:58:08.098053 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.097828    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-xks-ca\") pod \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\" (UID: \"10f720ee-ecfe-4e21-a2c3-ffe0519d901a\") "
Nov 03 17:58:08.101064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.101032    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-kube-api-access-bvzrw" pod "10f720ee-ecfe-4e21-a2c3-ffe0519d901a" (UID: "10f720ee-ecfe-4e21-a2c3-ffe0519d901a"). InnerVolumeSpecName "kube-api-access-bvzrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:08.101566 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.101540    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-openshift-ca" pod "10f720ee-ecfe-4e21-a2c3-ffe0519d901a" (UID: "10f720ee-ecfe-4e21-a2c3-ffe0519d901a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:08.104426 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.104391    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-xks-ca" pod "10f720ee-ecfe-4e21-a2c3-ffe0519d901a" (UID: "10f720ee-ecfe-4e21-a2c3-ffe0519d901a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:08.199544 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.199515    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvzrw\" (UniqueName: \"kubernetes.io/projected/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-kube-api-access-bvzrw\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:08.199544 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.199541    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:08.199727 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.199552    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f720ee-ecfe-4e21-a2c3-ffe0519d901a-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:08.809270 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.809239    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"
Nov 03 17:58:08.809270 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.809247    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h" event={"ID":"10f720ee-ecfe-4e21-a2c3-ffe0519d901a","Type":"ContainerDied","Data":"cecdb3ceebce130ab980de9a7633573837ea9ab7456a966fcf965a4acfefe8f7"}
Nov 03 17:58:08.809698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:08.809305    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cecdb3ceebce130ab980de9a7633573837ea9ab7456a966fcf965a4acfefe8f7"
Nov 03 17:58:09.719525 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.719494    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"]
Nov 03 17:58:09.720140 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720118    4493 cpu_manager.go:414] "containerMap: removing container" podUID="890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" containerName="revoke-keys"
Nov 03 17:58:09.720140 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720140    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" containerName="revoke-keys"
Nov 03 17:58:09.720315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720171    4493 cpu_manager.go:414] "containerMap: removing container" podUID="14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" containerName="revoke-keys"
Nov 03 17:58:09.720315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720178    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" containerName="revoke-keys"
Nov 03 17:58:09.720315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720199    4493 cpu_manager.go:414] "containerMap: removing container" podUID="10f720ee-ecfe-4e21-a2c3-ffe0519d901a" containerName="revoke-keys"
Nov 03 17:58:09.720315 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720206    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f720ee-ecfe-4e21-a2c3-ffe0519d901a" containerName="revoke-keys"
Nov 03 17:58:09.720504 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720400    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" containerName="revoke-keys"
Nov 03 17:58:09.720504 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720419    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" containerName="revoke-keys"
Nov 03 17:58:09.720504 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.720431    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f720ee-ecfe-4e21-a2c3-ffe0519d901a" containerName="revoke-keys"
Nov 03 17:58:09.725089 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.725067    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820234 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820202    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzpdx\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-kube-api-access-dzpdx\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820250    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820287    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820304    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820359    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820375    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820430    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820455    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded41810-cf11-46b9-8c36-f3a7f56647ef-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.820633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.820520    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.837474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.837446    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"]
Nov 03 17:58:09.901495 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.901461    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"]
Nov 03 17:58:09.906506 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.906485    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:09.921717 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.921691    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.921864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.921764    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzpdx\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-kube-api-access-dzpdx\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.921864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.921797    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.921864 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.921846    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922231    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922367 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922333    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922367 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922346    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922367 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922386    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922608 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922430    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922608 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922518    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922608 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922560    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded41810-cf11-46b9-8c36-f3a7f56647ef-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.922759 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.922739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.923153 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.923129    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.923635 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.923397    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded41810-cf11-46b9-8c36-f3a7f56647ef-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.927205 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.927170    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.928118 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.928093    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.938139 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.938114    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"]
Nov 03 17:58:09.943023 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.942999    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:09.971560 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.971201    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"]
Nov 03 17:58:09.992474 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.992444    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:09.993284 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:09.993246    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzpdx\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-kube-api-access-dzpdx\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:10.019129 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.019099    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"]
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025532    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025585    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025634    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025711    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025783    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be37b81-fe72-47b6-abfd-fb0f6933beac-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025814    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025865    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.025986    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026020    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026064    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026102    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026140    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026223    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vskl\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-kube-api-access-8vskl\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026256    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zt95\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-kube-api-access-2zt95\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026291    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.026323    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.027177    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.027249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.027220    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.040258 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.039968    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:10.129295 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129264    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129301    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129325    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129343    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129363    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129420    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vskl\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-kube-api-access-8vskl\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129451 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129439    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zt95\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-kube-api-access-2zt95\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129457    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129477    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129507    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129523    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129563    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129581    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129610    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129666    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129726    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be37b81-fe72-47b6-abfd-fb0f6933beac-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129753    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.129769 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129754    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.130335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.129826    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.130335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.130045    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.130335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.130288    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.130335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.130298    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.130558 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.130402    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.130558 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.130510    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.130558 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.130549    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.130768 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.130739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.131929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.131841    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be37b81-fe72-47b6-abfd-fb0f6933beac-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.132288 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.132269    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.136160 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.136107    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.136160 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.136122    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.136332 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.136285    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.137610 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.137590    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.195530 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.195464    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vskl\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-kube-api-access-8vskl\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.196031 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.195993    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.196378 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.196353    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zt95\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-kube-api-access-2zt95\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.197169 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.197148    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.221384 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.221358    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:10.268091 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.268056    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:10.493041 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:10.492999    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded41810_cf11_46b9_8c36_f3a7f56647ef.slice/crio-b5ffc95612ab6569df23bfce74c49c8c083a016150c698df748fe731ae14b9a9 WatchSource:0}: Error finding container b5ffc95612ab6569df23bfce74c49c8c083a016150c698df748fe731ae14b9a9: Status 404 returned error can't find the container with id b5ffc95612ab6569df23bfce74c49c8c083a016150c698df748fe731ae14b9a9
Nov 03 17:58:10.493768 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.493741    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"]
Nov 03 17:58:10.495989 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.495957    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.496113 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.496043    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.496113 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.496090    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.610977 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.610943    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"]
Nov 03 17:58:10.611202 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:10.611176    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be37b81_fe72_47b6_abfd_fb0f6933beac.slice/crio-2915fdbfd51145ce5d2389653566de31399dd68fb5716a0ddef314dc196b3ec9 WatchSource:0}: Error finding container 2915fdbfd51145ce5d2389653566de31399dd68fb5716a0ddef314dc196b3ec9: Status 404 returned error can't find the container with id 2915fdbfd51145ce5d2389653566de31399dd68fb5716a0ddef314dc196b3ec9
Nov 03 17:58:10.614978 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.614944    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.615077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.615015    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.615077 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.615059    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.679166 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.679135    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"]
Nov 03 17:58:10.679566 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:10.679539    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb5b51b_a416_4b17_a69a_9a3f32938a13.slice/crio-e25fb93dd3ec20b9d846d616344bcc4470664cd37e214cbb661a840e4feba2df WatchSource:0}: Error finding container e25fb93dd3ec20b9d846d616344bcc4470664cd37e214cbb661a840e4feba2df: Status 404 returned error can't find the container with id e25fb93dd3ec20b9d846d616344bcc4470664cd37e214cbb661a840e4feba2df
Nov 03 17:58:10.682258 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.682225    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.682356 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.682299    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.682356 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.682336    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:10.836075 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.836030    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" event={"ID":"fbb5b51b-a416-4b17-a69a-9a3f32938a13","Type":"ContainerStarted","Data":"72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075"}
Nov 03 17:58:10.836548 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.836104    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" event={"ID":"fbb5b51b-a416-4b17-a69a-9a3f32938a13","Type":"ContainerStarted","Data":"e25fb93dd3ec20b9d846d616344bcc4470664cd37e214cbb661a840e4feba2df"}
Nov 03 17:58:10.840891 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.840843    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" event={"ID":"5be37b81-fe72-47b6-abfd-fb0f6933beac","Type":"ContainerStarted","Data":"0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c"}
Nov 03 17:58:10.841021 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.840932    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" event={"ID":"5be37b81-fe72-47b6-abfd-fb0f6933beac","Type":"ContainerStarted","Data":"2915fdbfd51145ce5d2389653566de31399dd68fb5716a0ddef314dc196b3ec9"}
Nov 03 17:58:10.845683 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.844517    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" event={"ID":"ded41810-cf11-46b9-8c36-f3a7f56647ef","Type":"ContainerStarted","Data":"ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7"}
Nov 03 17:58:10.845683 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.844569    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" event={"ID":"ded41810-cf11-46b9-8c36-f3a7f56647ef","Type":"ContainerStarted","Data":"b5ffc95612ab6569df23bfce74c49c8c083a016150c698df748fe731ae14b9a9"}
Nov 03 17:58:10.963837 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:10.963776    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podStartSLOduration=1.963756907 podStartE2EDuration="1.963756907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:10.96215297 +0000 UTC m=+1470.421323957" watchObservedRunningTime="2026-11-03 17:58:10.963756907 +0000 UTC m=+1470.422927894"
Nov 03 17:58:11.040405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.040318    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:11.042447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.042414    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 17:58:11.042581 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.042479    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 17:58:11.092415 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.092365    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podStartSLOduration=2.092349825 podStartE2EDuration="2.092349825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:11.086688755 +0000 UTC m=+1470.545859742" watchObservedRunningTime="2026-11-03 17:58:11.092349825 +0000 UTC m=+1470.551520810"
Nov 03 17:58:11.224273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.224243    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:11.224521 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.224234    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podStartSLOduration=2.224210622 podStartE2EDuration="2.224210622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:11.211848402 +0000 UTC m=+1470.671019422" watchObservedRunningTime="2026-11-03 17:58:11.224210622 +0000 UTC m=+1470.683381611"
Nov 03 17:58:11.241896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.229435    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 17:58:11.241896 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.229494    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 17:58:11.268448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.268419    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:11.270585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.270548    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 17:58:11.271689 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.271656    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 17:58:11.696388 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.696352    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"]
Nov 03 17:58:11.714099 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.714065    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:11.714836 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.714791    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"]
Nov 03 17:58:11.797900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.771831    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:58:11.797900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.771933    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:58:11.797900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.772033    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:58:12.001949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:11.930389    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:58:12.011116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.011064    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jpf\" (UniqueName: \"kubernetes.io/projected/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-kube-api-access-d7jpf\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.011489 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.011467    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.011585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.011575    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.075902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.070010    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 17:58:12.075902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.070079    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 17:58:12.112867 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.112833    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.113686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.113664    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jpf\" (UniqueName: \"kubernetes.io/projected/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-kube-api-access-d7jpf\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.114136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.114114    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.114860 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.114837    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.173898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.164312    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.195441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.195405    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jpf\" (UniqueName: \"kubernetes.io/projected/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-kube-api-access-d7jpf\") pod \"maas-api-revoke-sub-3ec42eb2-45dlk\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:12.224025 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.223988    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 17:58:12.224758 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.224724    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 17:58:12.272431 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.272334    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 17:58:12.272707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.272679    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 17:58:12.314124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:12.314084    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:13.061899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:13.061578    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 17:58:13.061899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:13.061641    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 17:58:13.096163 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:13.096124    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"]
Nov 03 17:58:13.225231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:13.225197    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 17:58:13.225503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:13.225472    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 17:58:13.278576 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:13.278462    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 17:58:13.278576 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:13.278515    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 17:58:14.043901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.043509    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 17:58:14.043901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.043569    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 17:58:14.066899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.064551    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk" event={"ID":"eab632e2-f694-46a6-ac8c-df8c1f34b7dd","Type":"ContainerStarted","Data":"b045706cbd7eccccaa10e87daf893fcfb96e9661872622a6d51ca560bf4092c9"}
Nov 03 17:58:14.066899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.064625    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk" event={"ID":"eab632e2-f694-46a6-ac8c-df8c1f34b7dd","Type":"ContainerStarted","Data":"b5e4bf2124836572d2e935c638fbc8983db143bd755e047268b4807396e69b57"}
Nov 03 17:58:14.200405 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.200342    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk" podStartSLOduration=3.200322209 podStartE2EDuration="3.200322209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:14.194115435 +0000 UTC m=+1473.653286424" watchObservedRunningTime="2026-11-03 17:58:14.200322209 +0000 UTC m=+1473.659493201"
Nov 03 17:58:14.230903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.229410    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 17:58:14.230903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.229469    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 17:58:14.278899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.278060    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 17:58:14.278899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:14.278118    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 17:58:15.047292 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.047253    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 17:58:15.047585 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.047556    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 17:58:15.099330 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.099264    4493 generic.go:356] "Generic (PLEG): container finished" podID="eab632e2-f694-46a6-ac8c-df8c1f34b7dd" containerID="b045706cbd7eccccaa10e87daf893fcfb96e9661872622a6d51ca560bf4092c9" exitCode=0
Nov 03 17:58:15.099724 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.099343    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk" event={"ID":"eab632e2-f694-46a6-ac8c-df8c1f34b7dd","Type":"ContainerDied","Data":"b045706cbd7eccccaa10e87daf893fcfb96e9661872622a6d51ca560bf4092c9"}
Nov 03 17:58:15.236657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.236232    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 17:58:15.236657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.236290    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 17:58:15.294941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.294901    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 17:58:15.295201 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.295175    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 17:58:15.978905 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:15.863943    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"]
Nov 03 17:58:16.082111 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.045656    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 17:58:16.082111 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.045717    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 17:58:16.082111 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.052100    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"]
Nov 03 17:58:16.082111 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.052215    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.118917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.114959    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/e66e698d-7f14-43b2-8758-a09b1d16dbd3-kube-api-access-2zmbz\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.118917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.115200    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.118917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.115349    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.258288 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.258177    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.258686 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.258665    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.365859 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.278726    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 17:58:16.366242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.366105    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 17:58:16.379909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.280666    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.380541 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.380418    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/e66e698d-7f14-43b2-8758-a09b1d16dbd3-kube-api-access-2zmbz\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.396947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.290947    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 17:58:16.396947 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.392958    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 17:58:16.503898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.493466    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.590158 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.582645    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/e66e698d-7f14-43b2-8758-a09b1d16dbd3-kube-api-access-2zmbz\") pod \"maas-api-revoke-sub-e477a476-n8fm8\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:16.687080 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.687036    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:16.687417 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.687385    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:16.727256 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:16.727220    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:17.084910 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.084614    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Nov 03 17:58:17.084910 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.084675    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Nov 03 17:58:17.194891 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.194216    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:17.222685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.222364    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 17:58:17.222685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.222407    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 17:58:17.269407 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.269358    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Nov 03 17:58:17.269579 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.269420    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Nov 03 17:58:17.290938 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.290905    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk" event={"ID":"eab632e2-f694-46a6-ac8c-df8c1f34b7dd","Type":"ContainerDied","Data":"b5e4bf2124836572d2e935c638fbc8983db143bd755e047268b4807396e69b57"}
Nov 03 17:58:17.291199 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.291179    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e4bf2124836572d2e935c638fbc8983db143bd755e047268b4807396e69b57"
Nov 03 17:58:17.291376 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.291360    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"
Nov 03 17:58:17.341226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.341123    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-xks-ca\") pod \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") "
Nov 03 17:58:17.341226 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.341183    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-openshift-ca\") pod \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") "
Nov 03 17:58:17.348103 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.341352    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-kube-api-access-d7jpf\" (UniqueName: \"kubernetes.io/projected/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-kube-api-access-d7jpf\") pod \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\" (UID: \"eab632e2-f694-46a6-ac8c-df8c1f34b7dd\") "
Nov 03 17:58:17.348103 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.343162    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-openshift-ca" pod "eab632e2-f694-46a6-ac8c-df8c1f34b7dd" (UID: "eab632e2-f694-46a6-ac8c-df8c1f34b7dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:17.348103 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.347794    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-kube-api-access-d7jpf" pod "eab632e2-f694-46a6-ac8c-df8c1f34b7dd" (UID: "eab632e2-f694-46a6-ac8c-df8c1f34b7dd"). InnerVolumeSpecName "kube-api-access-d7jpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:17.350397 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.350357    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-xks-ca" pod "eab632e2-f694-46a6-ac8c-df8c1f34b7dd" (UID: "eab632e2-f694-46a6-ac8c-df8c1f34b7dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:17.436727 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:17.436686    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66e698d_7f14_43b2_8758_a09b1d16dbd3.slice/crio-50430f9023dd0dbf3f693be3bbc8e91fc2fcd2e3806be0974b3c600a82c5155e WatchSource:0}: Error finding container 50430f9023dd0dbf3f693be3bbc8e91fc2fcd2e3806be0974b3c600a82c5155e: Status 404 returned error can't find the container with id 50430f9023dd0dbf3f693be3bbc8e91fc2fcd2e3806be0974b3c600a82c5155e
Nov 03 17:58:17.437411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.437370    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"]
Nov 03 17:58:17.446116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.443392    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:17.446116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.443429    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jpf\" (UniqueName: \"kubernetes.io/projected/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-kube-api-access-d7jpf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:17.446116 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:17.443464    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eab632e2-f694-46a6-ac8c-df8c1f34b7dd-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:18.386740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.386678    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8" event={"ID":"e66e698d-7f14-43b2-8758-a09b1d16dbd3","Type":"ContainerStarted","Data":"a20054213ee984224b4cc6cd7772ca5a2668c2118074b0489380f3e6e3b3def3"}
Nov 03 17:58:18.387320 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.386785    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8" event={"ID":"e66e698d-7f14-43b2-8758-a09b1d16dbd3","Type":"ContainerStarted","Data":"50430f9023dd0dbf3f693be3bbc8e91fc2fcd2e3806be0974b3c600a82c5155e"}
Nov 03 17:58:18.518598 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.518532    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8" podStartSLOduration=3.5185089019999998 podStartE2EDuration="3.518508902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:18.514062085 +0000 UTC m=+1477.973233072" watchObservedRunningTime="2026-11-03 17:58:18.518508902 +0000 UTC m=+1477.977679892"
Nov 03 17:58:18.727090 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.727060    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:18.727665 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.727642    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:18.728705 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.728685    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 17:58:18.793069 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.793034    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:18.793429 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.793407    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:18.794661 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.794633    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 17:58:18.823025 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.822994    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:18.823348 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.823318    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:18.824242 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:18.824223    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 17:58:19.397456 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:19.397422    4493 generic.go:356] "Generic (PLEG): container finished" podID="e66e698d-7f14-43b2-8758-a09b1d16dbd3" containerID="a20054213ee984224b4cc6cd7772ca5a2668c2118074b0489380f3e6e3b3def3" exitCode=0
Nov 03 17:58:19.397942 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:19.397509    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8" event={"ID":"e66e698d-7f14-43b2-8758-a09b1d16dbd3","Type":"ContainerDied","Data":"a20054213ee984224b4cc6cd7772ca5a2668c2118074b0489380f3e6e3b3def3"}
Nov 03 17:58:20.561069 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.561044    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:20.595746 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.595712    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-xks-ca\") pod \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") "
Nov 03 17:58:20.595929 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.595841    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66e698d-7f14-43b2-8758-a09b1d16dbd3-kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/e66e698d-7f14-43b2-8758-a09b1d16dbd3-kube-api-access-2zmbz\") pod \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") "
Nov 03 17:58:20.596010 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.595989    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-openshift-ca\") pod \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\" (UID: \"e66e698d-7f14-43b2-8758-a09b1d16dbd3\") "
Nov 03 17:58:20.596309 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.596284    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-openshift-ca" pod "e66e698d-7f14-43b2-8758-a09b1d16dbd3" (UID: "e66e698d-7f14-43b2-8758-a09b1d16dbd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:20.596634 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.596608    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:20.598624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.598599    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-xks-ca" pod "e66e698d-7f14-43b2-8758-a09b1d16dbd3" (UID: "e66e698d-7f14-43b2-8758-a09b1d16dbd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:20.598624 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.598615    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66e698d-7f14-43b2-8758-a09b1d16dbd3-kube-api-access-2zmbz" pod "e66e698d-7f14-43b2-8758-a09b1d16dbd3" (UID: "e66e698d-7f14-43b2-8758-a09b1d16dbd3"). InnerVolumeSpecName "kube-api-access-2zmbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:20.697958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.697851    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66e698d-7f14-43b2-8758-a09b1d16dbd3-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:20.697958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:20.697911    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmbz\" (UniqueName: \"kubernetes.io/projected/e66e698d-7f14-43b2-8758-a09b1d16dbd3-kube-api-access-2zmbz\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:21.416957 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:21.416929    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"
Nov 03 17:58:21.416957 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:21.416936    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8" event={"ID":"e66e698d-7f14-43b2-8758-a09b1d16dbd3","Type":"ContainerDied","Data":"50430f9023dd0dbf3f693be3bbc8e91fc2fcd2e3806be0974b3c600a82c5155e"}
Nov 03 17:58:21.417227 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:21.416987    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50430f9023dd0dbf3f693be3bbc8e91fc2fcd2e3806be0974b3c600a82c5155e"
Nov 03 17:58:22.418347 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.418318    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"]
Nov 03 17:58:22.418888 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.418861    4493 cpu_manager.go:414] "containerMap: removing container" podUID="eab632e2-f694-46a6-ac8c-df8c1f34b7dd" containerName="revoke-keys"
Nov 03 17:58:22.418939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.418892    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab632e2-f694-46a6-ac8c-df8c1f34b7dd" containerName="revoke-keys"
Nov 03 17:58:22.418939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.418924    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e66e698d-7f14-43b2-8758-a09b1d16dbd3" containerName="revoke-keys"
Nov 03 17:58:22.418939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.418932    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66e698d-7f14-43b2-8758-a09b1d16dbd3" containerName="revoke-keys"
Nov 03 17:58:22.419079 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.419068    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66e698d-7f14-43b2-8758-a09b1d16dbd3" containerName="revoke-keys"
Nov 03 17:58:22.419113 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.419086    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab632e2-f694-46a6-ac8c-df8c1f34b7dd" containerName="revoke-keys"
Nov 03 17:58:22.422917 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.422895    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.482195 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.482167    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"]
Nov 03 17:58:22.521606 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.521562    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrkdg\" (UniqueName: \"kubernetes.io/projected/7b0f0c85-2e01-4005-9132-85df6d047cdd-kube-api-access-lrkdg\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.521886 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.521848    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.521964 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.521908    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.527522 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.527498    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:58:22.527522 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.527513    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:58:22.527689 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.527500    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:58:22.527689 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.527570    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:58:22.624181 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.624152    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.624374 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.624234    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrkdg\" (UniqueName: \"kubernetes.io/projected/7b0f0c85-2e01-4005-9132-85df6d047cdd-kube-api-access-lrkdg\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.624444 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.624409    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.624993 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.624972    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.626500 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.626481    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.688423 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.688345    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrkdg\" (UniqueName: \"kubernetes.io/projected/7b0f0c85-2e01-4005-9132-85df6d047cdd-kube-api-access-lrkdg\") pod \"maas-api-revoke-sub-18f6ea29-btxcn\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:22.736403 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:22.736375    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:23.123857 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:23.123825    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0f0c85_2e01_4005_9132_85df6d047cdd.slice/crio-4f033ab0e25bbfca7798130ec47fa48289f16996c4237e58f7d7b157cdb321ec WatchSource:0}: Error finding container 4f033ab0e25bbfca7798130ec47fa48289f16996c4237e58f7d7b157cdb321ec: Status 404 returned error can't find the container with id 4f033ab0e25bbfca7798130ec47fa48289f16996c4237e58f7d7b157cdb321ec
Nov 03 17:58:23.124631 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:23.124606    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"]
Nov 03 17:58:23.462909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:23.462831    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn" event={"ID":"7b0f0c85-2e01-4005-9132-85df6d047cdd","Type":"ContainerStarted","Data":"4f033ab0e25bbfca7798130ec47fa48289f16996c4237e58f7d7b157cdb321ec"}
Nov 03 17:58:24.472131 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:24.472102    4493 generic.go:356] "Generic (PLEG): container finished" podID="7b0f0c85-2e01-4005-9132-85df6d047cdd" containerID="a3d6e78a1d08f1192f29e87079cac52bba11a07f6126666d636f7902d25ae24d" exitCode=0
Nov 03 17:58:24.472543 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:24.472182    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn" event={"ID":"7b0f0c85-2e01-4005-9132-85df6d047cdd","Type":"ContainerDied","Data":"a3d6e78a1d08f1192f29e87079cac52bba11a07f6126666d636f7902d25ae24d"}
Nov 03 17:58:25.643413 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.643385    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:25.675513 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.675477    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-openshift-ca\") pod \"7b0f0c85-2e01-4005-9132-85df6d047cdd\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") "
Nov 03 17:58:25.675671 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.675649    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-xks-ca\") pod \"7b0f0c85-2e01-4005-9132-85df6d047cdd\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") "
Nov 03 17:58:25.675749 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.675700    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0f0c85-2e01-4005-9132-85df6d047cdd-kube-api-access-lrkdg\" (UniqueName: \"kubernetes.io/projected/7b0f0c85-2e01-4005-9132-85df6d047cdd-kube-api-access-lrkdg\") pod \"7b0f0c85-2e01-4005-9132-85df6d047cdd\" (UID: \"7b0f0c85-2e01-4005-9132-85df6d047cdd\") "
Nov 03 17:58:25.675813 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.675785    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-openshift-ca" pod "7b0f0c85-2e01-4005-9132-85df6d047cdd" (UID: "7b0f0c85-2e01-4005-9132-85df6d047cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:25.676214 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.676194    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:25.677911 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.677867    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-xks-ca" pod "7b0f0c85-2e01-4005-9132-85df6d047cdd" (UID: "7b0f0c85-2e01-4005-9132-85df6d047cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:25.678109 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.678086    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0f0c85-2e01-4005-9132-85df6d047cdd-kube-api-access-lrkdg" pod "7b0f0c85-2e01-4005-9132-85df6d047cdd" (UID: "7b0f0c85-2e01-4005-9132-85df6d047cdd"). InnerVolumeSpecName "kube-api-access-lrkdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:25.778312 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.778229    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0f0c85-2e01-4005-9132-85df6d047cdd-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:25.778312 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:25.778263    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrkdg\" (UniqueName: \"kubernetes.io/projected/7b0f0c85-2e01-4005-9132-85df6d047cdd-kube-api-access-lrkdg\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:26.489147 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:26.489117    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"
Nov 03 17:58:26.489335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:26.489147    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn" event={"ID":"7b0f0c85-2e01-4005-9132-85df6d047cdd","Type":"ContainerDied","Data":"4f033ab0e25bbfca7798130ec47fa48289f16996c4237e58f7d7b157cdb321ec"}
Nov 03 17:58:26.489335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:26.489196    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f033ab0e25bbfca7798130ec47fa48289f16996c4237e58f7d7b157cdb321ec"
Nov 03 17:58:28.499900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.496983    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.544895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.543979    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.622898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.619142    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.622898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.620085    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.643898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.641957    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 17:58:28.673900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.669964    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.782958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.781503    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:28.804901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:28.801764    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.217788 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.214980    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.217788 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.216223    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.217788 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.216927    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.279826 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.279052    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.341231 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.340464    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.372820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.372005    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 17:58:29.501088 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.500973    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.502082 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.502057    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:58:29.583932 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.561976    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.583932 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.562467    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:58:29.641038 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.628961    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:58:29.641038 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.629475    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:58:29.720900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.665047    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.720900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.665467    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:58:29.720900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.666371    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.720900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.666693    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:58:29.720900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.673322    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.720900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.673669    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:58:29.781611 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.781451    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded"
Nov 03 17:58:29.781853 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.781827    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:29.783079 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.782204    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:58:29.891388 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.861658    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:58:29.891388 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:29.863125    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:58:30.220447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.217974    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.220447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.218411    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:58:30.220447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.218598    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:58:30.220447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.218805    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:58:30.220447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.218861    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:58:30.220447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.218922    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:58:30.309901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.299245    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.311935 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.310896    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:58:30.363773 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.363730    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.400901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.368078    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:58:30.400901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.375176    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.400901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.376564    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:58:30.507621 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.507523    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.586901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.568478    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:58:30.705039 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.704992    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.773958 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.748491    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 17:58:30.802829 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.762955    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:58:30.817190 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.763004    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 17:58:30.830900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.787997    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 17:58:30.830900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.802000    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:30.830900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.829809    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.887902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.887244    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.957900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.932441    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:30.958172 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.934980    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:30.958359 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:30.958330    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.185063 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.184963    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.230090 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.230043    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.238906 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.230628    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.239905 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.239467    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.329899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.300269    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.408901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.385963    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.408901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.388549    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.408901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.388910    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 17:58:31.536228 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.536182    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.594895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.590432    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.684194 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.683159    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:58:31.706502 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.706340    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 17:58:31.807302 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.805651    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:58:31.807302 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.806046    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 17:58:31.826721 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.826675    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.827476 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.827430    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:31.975897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:31.951161    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.090968 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.068730    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:58:32.245696 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.235040    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.300897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.242977    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.300897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.253192    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.343131 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.343036    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.412263 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.411769    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.446067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.411804    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 17:58:32.602403 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.601988    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.755912 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.754967    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.868950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.816176    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.868950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.846457    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:32.868950 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:32.846728    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 17:58:33.157981 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.150824    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 17:58:33.228900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.224399    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 17:58:33.228900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.224482    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:58:33.228900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.227556    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:33.228900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.227605    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.246129 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.246085    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 03 17:58:33.285057 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.284997    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.285662 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.285622    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.425849 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.425752    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.428789 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.426098    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.447497 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.447457    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.597729 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.597687    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.622224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.611167    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.800523 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.800490    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:58:33.823158 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.823135    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:58:33.831208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.820383    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:33.893952 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.893917    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:58:33.956895 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:33.952849    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.061260 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.060682    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:58:34.161241 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.160076    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.332119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.307949    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.332119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.312651    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.332119 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.312777    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:58:34.413308 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.413279    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 17:58:34.416488 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.416464    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:58:34.437021 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.436981    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.449243 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.449215    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:58:34.449593 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.449555    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 17:58:34.449685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.449618    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:34.453941 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.453921    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:58:34.475971 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.475942    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:58:34.496373 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:34.496053    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:58:36.544211 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.544170    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"]
Nov 03 17:58:36.545006 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.544990    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7b0f0c85-2e01-4005-9132-85df6d047cdd" containerName="revoke-keys"
Nov 03 17:58:36.545076 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.545009    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0f0c85-2e01-4005-9132-85df6d047cdd" containerName="revoke-keys"
Nov 03 17:58:36.545203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.545192    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0f0c85-2e01-4005-9132-85df6d047cdd" containerName="revoke-keys"
Nov 03 17:58:36.555688 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.555651    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.611365 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.611333    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"]
Nov 03 17:58:36.613499 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.613476    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:58:36.652961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.652936    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:58:36.652961 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.652966    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:58:36.656546 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.656524    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njd2h\" (UniqueName: \"kubernetes.io/projected/c22da52e-9360-4ca9-b465-9d2a502f8281-kube-api-access-njd2h\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.656653 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.656576    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.656846 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.656818    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.664114 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.664097    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:58:36.668770 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.668751    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"]
Nov 03 17:58:36.673510 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.673496    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.693720 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.693695    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"]
Nov 03 17:58:36.758503 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.758469    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njd2h\" (UniqueName: \"kubernetes.io/projected/c22da52e-9360-4ca9-b465-9d2a502f8281-kube-api-access-njd2h\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.758679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.758519    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.758679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.758542    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.758679 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.758577    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.758781 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.758707    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7q7\" (UniqueName: \"kubernetes.io/projected/a1646af9-12a5-4a66-8e23-89435dc506dc-kube-api-access-pz7q7\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.758865 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.758842    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.759219 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.759202    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.761303 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.761285    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-xks-ca\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.822797 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.822711    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njd2h\" (UniqueName: \"kubernetes.io/projected/c22da52e-9360-4ca9-b465-9d2a502f8281-kube-api-access-njd2h\") pod \"maas-api-revoke-sub-238b4a52-xrjxt\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.860152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.860125    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7q7\" (UniqueName: \"kubernetes.io/projected/a1646af9-12a5-4a66-8e23-89435dc506dc-kube-api-access-pz7q7\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.860317 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.860284    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.860363 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.860326    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.861121 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.861062    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.862777 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.862756    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.870807 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.870790    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:36.924747 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.924718    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7q7\" (UniqueName: \"kubernetes.io/projected/a1646af9-12a5-4a66-8e23-89435dc506dc-kube-api-access-pz7q7\") pod \"maas-api-revoke-sub-d01cf4e8-n5klw\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:36.995927 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:36.987818    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:37.047638 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:58:37.047576    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:58:37.264011 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:37.263983    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22da52e_9360_4ca9_b465_9d2a502f8281.slice/crio-9c4de27cc60e83331cca357d5b1cee58cff4daa6e30404a007f2e9deda437537 WatchSource:0}: Error finding container 9c4de27cc60e83331cca357d5b1cee58cff4daa6e30404a007f2e9deda437537: Status 404 returned error can't find the container with id 9c4de27cc60e83331cca357d5b1cee58cff4daa6e30404a007f2e9deda437537
Nov 03 17:58:37.267566 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:37.267544    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"]
Nov 03 17:58:37.422858 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:37.422830    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"]
Nov 03 17:58:37.710749 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:37.710656    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw" event={"ID":"a1646af9-12a5-4a66-8e23-89435dc506dc","Type":"ContainerStarted","Data":"fe844d3b1cda226e6ca0b575e0a9d83fc9fade7531728e396ddf54375b592ffd"}
Nov 03 17:58:37.712976 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:37.712944    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt" event={"ID":"c22da52e-9360-4ca9-b465-9d2a502f8281","Type":"ContainerStarted","Data":"d6bc7a7da3e4637a2efea7c8ec00624231d895d4ece14e39fdaba51c15927bb2"}
Nov 03 17:58:37.713120 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:37.713001    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt" event={"ID":"c22da52e-9360-4ca9-b465-9d2a502f8281","Type":"ContainerStarted","Data":"9c4de27cc60e83331cca357d5b1cee58cff4daa6e30404a007f2e9deda437537"}
Nov 03 17:58:37.843899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:37.839818    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt" podStartSLOduration=1.8397989190000001 podStartE2EDuration="1.839798919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:37.835589284 +0000 UTC m=+1497.294760292" watchObservedRunningTime="2026-11-03 17:58:37.839798919 +0000 UTC m=+1497.298969906"
Nov 03 17:58:38.732382 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:38.732339    4493 generic.go:356] "Generic (PLEG): container finished" podID="c22da52e-9360-4ca9-b465-9d2a502f8281" containerID="d6bc7a7da3e4637a2efea7c8ec00624231d895d4ece14e39fdaba51c15927bb2" exitCode=0
Nov 03 17:58:38.732890 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:38.732719    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt" event={"ID":"c22da52e-9360-4ca9-b465-9d2a502f8281","Type":"ContainerDied","Data":"d6bc7a7da3e4637a2efea7c8ec00624231d895d4ece14e39fdaba51c15927bb2"}
Nov 03 17:58:38.736005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:38.735974    4493 generic.go:356] "Generic (PLEG): container finished" podID="a1646af9-12a5-4a66-8e23-89435dc506dc" containerID="e563c6bf27b4990a541c17fd5686fa6e4b41691f366abdea80306a21ea6adb33" exitCode=0
Nov 03 17:58:38.736126 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:38.736072    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw" event={"ID":"a1646af9-12a5-4a66-8e23-89435dc506dc","Type":"ContainerDied","Data":"e563c6bf27b4990a541c17fd5686fa6e4b41691f366abdea80306a21ea6adb33"}
Nov 03 17:58:39.904253 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:39.904221    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:39.948106 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:39.948084    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:40.026660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.026584    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-openshift-ca\") pod \"c22da52e-9360-4ca9-b465-9d2a502f8281\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") "
Nov 03 17:58:40.026660 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.026658    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-openshift-ca\") pod \"a1646af9-12a5-4a66-8e23-89435dc506dc\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") "
Nov 03 17:58:40.027422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.027043    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-openshift-ca" pod "a1646af9-12a5-4a66-8e23-89435dc506dc" (UID: "a1646af9-12a5-4a66-8e23-89435dc506dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:40.027422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.027055    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22da52e-9360-4ca9-b465-9d2a502f8281-kube-api-access-njd2h\" (UniqueName: \"kubernetes.io/projected/c22da52e-9360-4ca9-b465-9d2a502f8281-kube-api-access-njd2h\") pod \"c22da52e-9360-4ca9-b465-9d2a502f8281\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") "
Nov 03 17:58:40.027422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.027067    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-openshift-ca" pod "c22da52e-9360-4ca9-b465-9d2a502f8281" (UID: "c22da52e-9360-4ca9-b465-9d2a502f8281"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:40.027422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.027130    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-xks-ca\") pod \"c22da52e-9360-4ca9-b465-9d2a502f8281\" (UID: \"c22da52e-9360-4ca9-b465-9d2a502f8281\") "
Nov 03 17:58:40.027422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.027271    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1646af9-12a5-4a66-8e23-89435dc506dc-kube-api-access-pz7q7\" (UniqueName: \"kubernetes.io/projected/a1646af9-12a5-4a66-8e23-89435dc506dc-kube-api-access-pz7q7\") pod \"a1646af9-12a5-4a66-8e23-89435dc506dc\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") "
Nov 03 17:58:40.027422 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.027312    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-xks-ca\") pod \"a1646af9-12a5-4a66-8e23-89435dc506dc\" (UID: \"a1646af9-12a5-4a66-8e23-89435dc506dc\") "
Nov 03 17:58:40.028243 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.028218    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:40.028377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.028248    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:40.029860 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.029832    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-xks-ca" pod "c22da52e-9360-4ca9-b465-9d2a502f8281" (UID: "c22da52e-9360-4ca9-b465-9d2a502f8281"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:40.030138 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.030109    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22da52e-9360-4ca9-b465-9d2a502f8281-kube-api-access-njd2h" pod "c22da52e-9360-4ca9-b465-9d2a502f8281" (UID: "c22da52e-9360-4ca9-b465-9d2a502f8281"). InnerVolumeSpecName "kube-api-access-njd2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:40.030243 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.030166    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1646af9-12a5-4a66-8e23-89435dc506dc-kube-api-access-pz7q7" pod "a1646af9-12a5-4a66-8e23-89435dc506dc" (UID: "a1646af9-12a5-4a66-8e23-89435dc506dc"). InnerVolumeSpecName "kube-api-access-pz7q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:40.030954 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.030933    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-xks-ca" pod "a1646af9-12a5-4a66-8e23-89435dc506dc" (UID: "a1646af9-12a5-4a66-8e23-89435dc506dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:40.129514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.129485    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22da52e-9360-4ca9-b465-9d2a502f8281-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:40.129514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.129509    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7q7\" (UniqueName: \"kubernetes.io/projected/a1646af9-12a5-4a66-8e23-89435dc506dc-kube-api-access-pz7q7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:40.129514 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.129519    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1646af9-12a5-4a66-8e23-89435dc506dc-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:40.129719 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.129529    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njd2h\" (UniqueName: \"kubernetes.io/projected/c22da52e-9360-4ca9-b465-9d2a502f8281-kube-api-access-njd2h\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:40.751557 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.751529    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"
Nov 03 17:58:40.751685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.751554    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw" event={"ID":"a1646af9-12a5-4a66-8e23-89435dc506dc","Type":"ContainerDied","Data":"fe844d3b1cda226e6ca0b575e0a9d83fc9fade7531728e396ddf54375b592ffd"}
Nov 03 17:58:40.751685 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.751599    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe844d3b1cda226e6ca0b575e0a9d83fc9fade7531728e396ddf54375b592ffd"
Nov 03 17:58:40.753013 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.752986    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt" event={"ID":"c22da52e-9360-4ca9-b465-9d2a502f8281","Type":"ContainerDied","Data":"9c4de27cc60e83331cca357d5b1cee58cff4daa6e30404a007f2e9deda437537"}
Nov 03 17:58:40.753125 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.753025    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"
Nov 03 17:58:40.753125 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:40.753034    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c4de27cc60e83331cca357d5b1cee58cff4daa6e30404a007f2e9deda437537"
Nov 03 17:58:41.119592 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.119516    4493 scope.go:122] "RemoveContainer" containerID="3862aa6cb1afeac2e3868aa071bf3d39ea21b2c7c79ba71887ca896e37637bba"
Nov 03 17:58:41.129918 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.129895    4493 scope.go:122] "RemoveContainer" containerID="be2d12b5a247cae046e24ed11ae0866be42b65c2148801df99eb90d9e6aecd62"
Nov 03 17:58:41.139037 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.139010    4493 scope.go:122] "RemoveContainer" containerID="b5b3d2e859e7fcfa7422ae246b7aec4809a7b8fe57c0e0ee2e025bb27fca4181"
Nov 03 17:58:41.150912 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.150891    4493 scope.go:122] "RemoveContainer" containerID="7123628d53469e4234a55967ba5f788ec2ea1b0d724fe1bb2d8347ac43793c53"
Nov 03 17:58:41.159968 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.159947    4493 scope.go:122] "RemoveContainer" containerID="532ca4b8a475f6676625637f4942f515cc316b07d462db78c1a3f5f6b8e4851e"
Nov 03 17:58:41.169176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.169159    4493 scope.go:122] "RemoveContainer" containerID="6f86a8979ce479824ed92ae3b047690e4352faea6d1c6a890a6b180d9e06b372"
Nov 03 17:58:41.177786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.177766    4493 scope.go:122] "RemoveContainer" containerID="3bb053f175f3dba2ec00247fee4118ac53ddae070c17ccfa3da7854eaac8dce3"
Nov 03 17:58:41.186847 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.186829    4493 scope.go:122] "RemoveContainer" containerID="e745926234ae02e5b80b5b1f0c926882c1f5c90bc1cab9da734fafcb48eceeec"
Nov 03 17:58:41.195174 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:41.195156    4493 scope.go:122] "RemoveContainer" containerID="2fd76b63ef326bbc5f3d9573de7425852410d9072d7df1a206db2bf441bff693"
Nov 03 17:58:42.094299 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.094268    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"]
Nov 03 17:58:42.094944 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.094923    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c22da52e-9360-4ca9-b465-9d2a502f8281" containerName="revoke-keys"
Nov 03 17:58:42.094944 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.094944    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22da52e-9360-4ca9-b465-9d2a502f8281" containerName="revoke-keys"
Nov 03 17:58:42.095127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.094984    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1646af9-12a5-4a66-8e23-89435dc506dc" containerName="revoke-keys"
Nov 03 17:58:42.095127 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.094992    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1646af9-12a5-4a66-8e23-89435dc506dc" containerName="revoke-keys"
Nov 03 17:58:42.095224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.095168    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1646af9-12a5-4a66-8e23-89435dc506dc" containerName="revoke-keys"
Nov 03 17:58:42.095224 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.095189    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22da52e-9360-4ca9-b465-9d2a502f8281" containerName="revoke-keys"
Nov 03 17:58:42.100794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.100772    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.179833 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.179795    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"]
Nov 03 17:58:42.260149 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260111    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-kube-api-access-fqj9s\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260255    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260319 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260297    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260350    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260378    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260454 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260418    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260604 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260492    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260604 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260522    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.260604 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.260571    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7908bb-be3a-473a-9179-e511506c7eb4-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.361799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.361724    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.361799 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.361773    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7908bb-be3a-473a-9179-e511506c7eb4-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.361827    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-kube-api-access-fqj9s\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.361952    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.361981    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362035 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362030    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362049    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362082    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362142    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362154    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362547 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362471    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362612 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362597    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362671 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362614    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7908bb-be3a-473a-9179-e511506c7eb4-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.362733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.362715    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.364435 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.364414    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.364673 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.364656    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.426475 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.426443    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.426635 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.426555    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-kube-api-access-fqj9s\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:42.715387 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:42.715356    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:43.155740 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:43.155711    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"]
Nov 03 17:58:43.155860 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:43.155708    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7908bb_be3a_473a_9179_e511506c7eb4.slice/crio-fc72f70f3de67a04b964539ee68a388adea136d28c8dd6526af94a0cfd29a03d WatchSource:0}: Error finding container fc72f70f3de67a04b964539ee68a388adea136d28c8dd6526af94a0cfd29a03d: Status 404 returned error can't find the container with id fc72f70f3de67a04b964539ee68a388adea136d28c8dd6526af94a0cfd29a03d
Nov 03 17:58:43.158399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:43.158367    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:43.158499 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:43.158434    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:43.158499 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:43.158461    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:43.778347 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:43.778312    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" event={"ID":"1c7908bb-be3a-473a-9179-e511506c7eb4","Type":"ContainerStarted","Data":"be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b"}
Nov 03 17:58:43.778748 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:43.778369    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" event={"ID":"1c7908bb-be3a-473a-9179-e511506c7eb4","Type":"ContainerStarted","Data":"fc72f70f3de67a04b964539ee68a388adea136d28c8dd6526af94a0cfd29a03d"}
Nov 03 17:58:44.008531 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:44.008480    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podStartSLOduration=2.008467248 podStartE2EDuration="2.008467248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:43.954626725 +0000 UTC m=+1503.413797712" watchObservedRunningTime="2026-11-03 17:58:44.008467248 +0000 UTC m=+1503.467638234"
Nov 03 17:58:44.720909 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:44.720492    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:44.740908 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:44.736362    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 17:58:44.740908 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:44.736421    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 17:58:45.718901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:45.718223    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 17:58:45.718901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:45.718281    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 17:58:46.723901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:46.719176    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 17:58:46.723901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:46.719234    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 17:58:47.447729 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.447695    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"]
Nov 03 17:58:47.453437 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.453415    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.459143 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.459113    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8sc\" (UniqueName: \"kubernetes.io/projected/fa899b9b-aa21-4fd6-8f52-ab44711e2996-kube-api-access-8l8sc\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.459264 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.459164    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.459347 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.459294    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.512057 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.512025    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"]
Nov 03 17:58:47.556795 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.556762    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 17:58:47.557102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.557063    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 17:58:47.557102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.557066    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 17:58:47.560399 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.560369    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8sc\" (UniqueName: \"kubernetes.io/projected/fa899b9b-aa21-4fd6-8f52-ab44711e2996-kube-api-access-8l8sc\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.560596 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.560581    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.560735 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.560720    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.561534 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.561513    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.563914 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.563893    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.572776 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.572743    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 17:58:47.579743 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.579719    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"]
Nov 03 17:58:47.585152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.585130    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.653067 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.653007    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"]
Nov 03 17:58:47.656270 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.656110    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8sc\" (UniqueName: \"kubernetes.io/projected/fa899b9b-aa21-4fd6-8f52-ab44711e2996-kube-api-access-8l8sc\") pod \"maas-api-revoke-sub-bb093433-587kf\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.662249 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.662161    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjk6l\" (UniqueName: \"kubernetes.io/projected/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-kube-api-access-rjk6l\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.662411 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.662390    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.662487 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.662461    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.721897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.717329    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 17:58:47.721897 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.717415    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 17:58:47.763445 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.763405    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.763828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.763486    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjk6l\" (UniqueName: \"kubernetes.io/projected/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-kube-api-access-rjk6l\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.763828 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.763669    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.764429 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.764405    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.766457 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.766431    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.771495 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.771470    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:47.843913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.843656    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjk6l\" (UniqueName: \"kubernetes.io/projected/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-kube-api-access-rjk6l\") pod \"maas-api-revoke-sub-e986bb5b-d7jfq\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:47.904555 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:47.904511    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:48.195087 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:48.195031    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"]
Nov 03 17:58:48.200795 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:48.198769    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa899b9b_aa21_4fd6_8f52_ab44711e2996.slice/crio-929e9f831d8391a647ca8aa1a469b7d1f6d4bace8e368571b5113628a7f8098f WatchSource:0}: Error finding container 929e9f831d8391a647ca8aa1a469b7d1f6d4bace8e368571b5113628a7f8098f: Status 404 returned error can't find the container with id 929e9f831d8391a647ca8aa1a469b7d1f6d4bace8e368571b5113628a7f8098f
Nov 03 17:58:48.351865 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:48.350592    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"]
Nov 03 17:58:48.756820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:48.756054    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 17:58:48.756820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:48.756132    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 17:58:48.868094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:48.867716    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf" event={"ID":"fa899b9b-aa21-4fd6-8f52-ab44711e2996","Type":"ContainerStarted","Data":"d1942bdc2f7abdf38fc3c773bf6f0194e10837cacb902cb6d62502043fee8e21"}
Nov 03 17:58:48.868094 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:48.867782    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf" event={"ID":"fa899b9b-aa21-4fd6-8f52-ab44711e2996","Type":"ContainerStarted","Data":"929e9f831d8391a647ca8aa1a469b7d1f6d4bace8e368571b5113628a7f8098f"}
Nov 03 17:58:48.872217 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:48.871954    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq" event={"ID":"c45cdf8f-e31c-46fb-b29a-7b82d23ced50","Type":"ContainerStarted","Data":"83267b20896c699d10acb3c20c2765002c980c1800c7d87dad61eb11e11d3106"}
Nov 03 17:58:49.002433 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:49.002379    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf" podStartSLOduration=2.002366512 podStartE2EDuration="2.002366512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:48.99464296 +0000 UTC m=+1508.453813940" watchObservedRunningTime="2026-11-03 17:58:49.002366512 +0000 UTC m=+1508.461537508"
Nov 03 17:58:49.715820 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:49.715784    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 17:58:49.716015 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:49.715849    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 17:58:49.881945 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:49.881910    4493 generic.go:356] "Generic (PLEG): container finished" podID="fa899b9b-aa21-4fd6-8f52-ab44711e2996" containerID="d1942bdc2f7abdf38fc3c773bf6f0194e10837cacb902cb6d62502043fee8e21" exitCode=0
Nov 03 17:58:49.882372 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:49.881955    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf" event={"ID":"fa899b9b-aa21-4fd6-8f52-ab44711e2996","Type":"ContainerDied","Data":"d1942bdc2f7abdf38fc3c773bf6f0194e10837cacb902cb6d62502043fee8e21"}
Nov 03 17:58:49.883865 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:49.883836    4493 generic.go:356] "Generic (PLEG): container finished" podID="c45cdf8f-e31c-46fb-b29a-7b82d23ced50" containerID="32eef1ed380dd5201922204eab72d86d990084cdd888c2f2dea0fcda95f6a9f1" exitCode=0
Nov 03 17:58:49.883998 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:49.883953    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq" event={"ID":"c45cdf8f-e31c-46fb-b29a-7b82d23ced50","Type":"ContainerDied","Data":"32eef1ed380dd5201922204eab72d86d990084cdd888c2f2dea0fcda95f6a9f1"}
Nov 03 17:58:50.723221 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:50.723183    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:50.728448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:50.724281    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:50.728448 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:50.725239    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 17:58:51.082748 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.082719    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:51.116736 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.116712    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:51.133118 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.133092    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa899b9b-aa21-4fd6-8f52-ab44711e2996-kube-api-access-8l8sc\" (UniqueName: \"kubernetes.io/projected/fa899b9b-aa21-4fd6-8f52-ab44711e2996-kube-api-access-8l8sc\") pod \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") "
Nov 03 17:58:51.133273 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.133128    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-openshift-ca\") pod \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") "
Nov 03 17:58:51.133344 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.133300    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-xks-ca\") pod \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\" (UID: \"fa899b9b-aa21-4fd6-8f52-ab44711e2996\") "
Nov 03 17:58:51.133567 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.133539    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-openshift-ca" pod "fa899b9b-aa21-4fd6-8f52-ab44711e2996" (UID: "fa899b9b-aa21-4fd6-8f52-ab44711e2996"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:51.134055 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.134035    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:51.135447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.135419    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa899b9b-aa21-4fd6-8f52-ab44711e2996-kube-api-access-8l8sc" pod "fa899b9b-aa21-4fd6-8f52-ab44711e2996" (UID: "fa899b9b-aa21-4fd6-8f52-ab44711e2996"). InnerVolumeSpecName "kube-api-access-8l8sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:51.135447 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.135426    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-xks-ca" pod "fa899b9b-aa21-4fd6-8f52-ab44711e2996" (UID: "fa899b9b-aa21-4fd6-8f52-ab44711e2996"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:51.235478 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.235424    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-kube-api-access-rjk6l\" (UniqueName: \"kubernetes.io/projected/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-kube-api-access-rjk6l\") pod \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") "
Nov 03 17:58:51.235669 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.235599    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-openshift-ca\") pod \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") "
Nov 03 17:58:51.235832 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.235802    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-xks-ca\") pod \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\" (UID: \"c45cdf8f-e31c-46fb-b29a-7b82d23ced50\") "
Nov 03 17:58:51.236441 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.236383    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-openshift-ca" pod "c45cdf8f-e31c-46fb-b29a-7b82d23ced50" (UID: "c45cdf8f-e31c-46fb-b29a-7b82d23ced50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 17:58:51.236808 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.236783    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8sc\" (UniqueName: \"kubernetes.io/projected/fa899b9b-aa21-4fd6-8f52-ab44711e2996-kube-api-access-8l8sc\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:51.236883 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.236819    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:51.236883 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.236835    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa899b9b-aa21-4fd6-8f52-ab44711e2996-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:51.237972 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.237949    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-kube-api-access-rjk6l" pod "c45cdf8f-e31c-46fb-b29a-7b82d23ced50" (UID: "c45cdf8f-e31c-46fb-b29a-7b82d23ced50"). InnerVolumeSpecName "kube-api-access-rjk6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 17:58:51.238082 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.238064    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-xks-ca" pod "c45cdf8f-e31c-46fb-b29a-7b82d23ced50" (UID: "c45cdf8f-e31c-46fb-b29a-7b82d23ced50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 17:58:51.338377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.338296    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:51.338377 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.338332    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjk6l\" (UniqueName: \"kubernetes.io/projected/c45cdf8f-e31c-46fb-b29a-7b82d23ced50-kube-api-access-rjk6l\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 17:58:51.904026 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.903996    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"
Nov 03 17:58:51.904209 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.904034    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf" event={"ID":"fa899b9b-aa21-4fd6-8f52-ab44711e2996","Type":"ContainerDied","Data":"929e9f831d8391a647ca8aa1a469b7d1f6d4bace8e368571b5113628a7f8098f"}
Nov 03 17:58:51.904209 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.904083    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929e9f831d8391a647ca8aa1a469b7d1f6d4bace8e368571b5113628a7f8098f"
Nov 03 17:58:51.907863 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.907840    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"
Nov 03 17:58:51.908012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.907864    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq" event={"ID":"c45cdf8f-e31c-46fb-b29a-7b82d23ced50","Type":"ContainerDied","Data":"83267b20896c699d10acb3c20c2765002c980c1800c7d87dad61eb11e11d3106"}
Nov 03 17:58:51.908012 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:51.907948    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83267b20896c699d10acb3c20c2765002c980c1800c7d87dad61eb11e11d3106"
Nov 03 17:58:55.064066 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.064010    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.176899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.164982    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.242026 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.241971    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.253008 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.252959    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.253707 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.253273    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.254611 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.253301    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:58:55.388953 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.386972    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 03 17:58:55.407901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.402966    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.407901 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.403365    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 17:58:55.573902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.548965    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.573902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.549082    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 17:58:55.625703 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.623983    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 17:58:55.726953 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.726908    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.767656 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.767612    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"]
Nov 03 17:58:55.768838 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.768806    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c45cdf8f-e31c-46fb-b29a-7b82d23ced50" containerName="revoke-keys"
Nov 03 17:58:55.779964 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.779925    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45cdf8f-e31c-46fb-b29a-7b82d23ced50" containerName="revoke-keys"
Nov 03 17:58:55.780208 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.780194    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fa899b9b-aa21-4fd6-8f52-ab44711e2996" containerName="revoke-keys"
Nov 03 17:58:55.780294 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.780282    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa899b9b-aa21-4fd6-8f52-ab44711e2996" containerName="revoke-keys"
Nov 03 17:58:55.780670 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.780646    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45cdf8f-e31c-46fb-b29a-7b82d23ced50" containerName="revoke-keys"
Nov 03 17:58:55.780822 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.780808    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa899b9b-aa21-4fd6-8f52-ab44711e2996" containerName="revoke-keys"
Nov 03 17:58:55.810148 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.810095    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 17:58:55.812449 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.812407    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:55.812698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.812667    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.838899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.827688    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:55.839509 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.829713    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"]
Nov 03 17:58:55.876301 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.876271    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"]
Nov 03 17:58:55.882105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.882008    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.017960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.996196    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.017960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.996275    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.017960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.996354    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.017960 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:55.996384    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038053    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-kube-api-access-bpxdg\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038139    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038219    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtt8f\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-kube-api-access-vtt8f\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038275    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038353    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038402    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038442    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038545    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038577    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038687    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038736    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038767    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038801    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.038828    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.059913 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.039199    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"]
Nov 03 17:58:56.079737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.072949    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.079737 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.073445    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:58:56.268175 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.268038    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.272907 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.268467    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.272907 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.272794    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:58:56.272907 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.272861    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:58:56.297070 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.269434    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.270146    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297422    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297537    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297585    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297616    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297649    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297677    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297739    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297783    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297839    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297865    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.297964    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-kube-api-access-bpxdg\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298021    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298104    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtt8f\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-kube-api-access-vtt8f\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298159    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298284    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298320    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298865    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.298994    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.299625    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.300324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.299972    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.343368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.323308    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.343368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.323582    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.343368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.323819    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.343368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.324100    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.343368 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.324746    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.371064 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.371026    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.424900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.371550    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.440021 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.439963    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.440560 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.440538    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:58:56.444908 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.440726    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.444908 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.442259    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:58:56.537511 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.509193    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.562924 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.552132    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.570455 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.563479    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:58:56.574184 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.574151    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtt8f\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-kube-api-access-vtt8f\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.626939 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.594120    4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded"
Nov 03 17:58:56.627507 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.605605    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-kube-api-access-bpxdg\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.627786 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.627249    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.641934 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.628126    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:58:56.774018 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.686653    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.817954 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.817358    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.838604 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.838572    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.870900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.842560    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.874136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.843138    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:56.874136 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.857362    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.894521    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.859808    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.923953    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.924015    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.924052    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.924081    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.924117    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.924155    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.924189    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:58:56.952902 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.924219    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 17:58:56.974427 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.974299    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:56.991733 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:56.988938    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:57.038898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.035384    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 17:58:57.042903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.039462    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 17:58:57.084572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.083271    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:57.103594 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.103471    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 17:58:57.122442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.105095    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 17:58:57.122442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.106296    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 17:58:57.122442 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.107678    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 17:58:57.129176 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.129151    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 17:58:57.148899 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.146364    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 17:58:57.663515 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:58:57.663480    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dcf183_0ab9_44f5_a9bc_6989be08e24c.slice/crio-f4e93fca17b68ae62a1157487336b2dc0ef0654ed82c53132d9306b09fa5464f WatchSource:0}: Error finding container f4e93fca17b68ae62a1157487336b2dc0ef0654ed82c53132d9306b09fa5464f: Status 404 returned error can't find the container with id f4e93fca17b68ae62a1157487336b2dc0ef0654ed82c53132d9306b09fa5464f
Nov 03 17:58:57.664211 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.664189    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"]
Nov 03 17:58:57.666802 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.666755    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:57.667005 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.666978    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:57.667174 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.667151    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:57.671845 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.671813    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"]
Nov 03 17:58:57.673533 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.673505    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:57.673633 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.673580    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:57.673726 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:57.673632    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:58:58.302714 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:58.302637    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" event={"ID":"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7","Type":"ContainerStarted","Data":"6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246"}
Nov 03 17:58:58.303253 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:58.302734    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" event={"ID":"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7","Type":"ContainerStarted","Data":"7baf1e754c174196c896f88ae37de5c66d7ffadd78b473ac64ad249bda1cf84a"}
Nov 03 17:58:58.305467 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:58.305437    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" event={"ID":"11dcf183-0ab9-44f5-a9bc-6989be08e24c","Type":"ContainerStarted","Data":"9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b"}
Nov 03 17:58:58.305622 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:58.305492    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" event={"ID":"11dcf183-0ab9-44f5-a9bc-6989be08e24c","Type":"ContainerStarted","Data":"f4e93fca17b68ae62a1157487336b2dc0ef0654ed82c53132d9306b09fa5464f"}
Nov 03 17:58:59.001898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:58.992284    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:58:59.001898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:58.994010    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 17:58:59.001898 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:58.994060    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 17:58:59.095335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:59.095303    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:58:59.097344 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:59.097318    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 17:58:59.097599 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:59.097557    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 17:58:59.503340 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:59.503285    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podStartSLOduration=4.503270144 podStartE2EDuration="4.503270144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:59.500819496 +0000 UTC m=+1518.959990478" watchObservedRunningTime="2026-11-03 17:58:59.503270144 +0000 UTC m=+1518.962441129"
Nov 03 17:58:59.647903 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:59.644778    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podStartSLOduration=4.644758684 podStartE2EDuration="4.644758684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:59.642395225 +0000 UTC m=+1519.101566213" watchObservedRunningTime="2026-11-03 17:58:59.644758684 +0000 UTC m=+1519.103929671"
Nov 03 17:58:59.989698 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:59.989662    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 17:58:59.989900 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:58:59.989714    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 17:59:00.089335 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:00.089306    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 17:59:00.089572 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:00.089549    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 17:59:00.990385 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:00.990337    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 17:59:00.990762 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:00.990397    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 17:59:01.090156 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:01.090123    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 17:59:01.090326 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:01.090184    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 17:59:01.995038 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:01.994996    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 17:59:01.995516 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:01.995058    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 17:59:02.091930 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:02.091472    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 17:59:02.091930 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:02.091531    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 17:59:02.989980 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:02.989937    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 17:59:02.990185 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:02.989997    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 17:59:03.090171 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:03.090128    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 17:59:03.090547 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:03.090189    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 17:59:03.990711 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:03.990672    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 17:59:03.991132 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:03.991078    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 17:59:04.095482 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:04.095445    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 17:59:04.096124 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:04.096091    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 17:59:04.990653 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:04.990620    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 17:59:04.990813 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:04.990681    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 17:59:05.089964 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:05.089928    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 17:59:05.090141 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:05.090007    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 17:59:05.996197 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:05.996162    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:59:05.997233 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:05.997158    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:59:05.998048 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:05.998016    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 17:59:06.101203 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:06.101171    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:59:06.101508 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:06.101488    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:59:06.102529 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:06.102510    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 17:59:07.004472 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:59:07.004435    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:59:10.069490 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.069456    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"]
Nov 03 17:59:10.074152 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.074128    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.156936 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.156899    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157123 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157071    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157193 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157125    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157247 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157221    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157306 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157286    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzl6\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-kube-api-access-ztzl6\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157357 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157340    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157420 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157373    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157477 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157417    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b680c-1d27-42b4-b971-8377d277d16f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.157477 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.157444    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.259591 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.259553    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzl6\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-kube-api-access-ztzl6\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.259771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.259631    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.259771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.259661    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.259771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.259691    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b680c-1d27-42b4-b971-8377d277d16f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.259771 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.259717    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260040 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.259918    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260040    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260102 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260086    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260209 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260159    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260209 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260192    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260311 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260222    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260396 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260371    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.260794 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.260781    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b680c-1d27-42b4-b971-8377d277d16f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.262657 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.262606    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.262782 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.262755    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.291125 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.291057    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"]
Nov 03 17:59:10.325105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.324916    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.325105 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.325039    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzl6\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-kube-api-access-ztzl6\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tlz9j\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.389347 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.389319    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:10.837440 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.837412    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"]
Nov 03 17:59:10.870816 ip-10-0-131-172 kubenswrapper[4493]: W1103 17:59:10.870787    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b680c_1d27_42b4_b971_8377d277d16f.slice/crio-287f4668491e29cff8897d9ecf55804d9d94fdcb666f7149ad55ebeb69b210e8 WatchSource:0}: Error finding container 287f4668491e29cff8897d9ecf55804d9d94fdcb666f7149ad55ebeb69b210e8: Status 404 returned error can't find the container with id 287f4668491e29cff8897d9ecf55804d9d94fdcb666f7149ad55ebeb69b210e8
Nov 03 17:59:10.873348 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.873312    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:10.873460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.873383    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:10.873460 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:10.873425    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 17:59:11.564003 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:11.563964    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" event={"ID":"7e0b680c-1d27-42b4-b971-8377d277d16f","Type":"ContainerStarted","Data":"000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b"}
Nov 03 17:59:11.564473 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:11.564057    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" event={"ID":"7e0b680c-1d27-42b4-b971-8377d277d16f","Type":"ContainerStarted","Data":"287f4668491e29cff8897d9ecf55804d9d94fdcb666f7149ad55ebeb69b210e8"}
Nov 03 17:59:11.756739 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:11.756683    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podStartSLOduration=1.756668247 podStartE2EDuration="1.756668247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:11.753123571 +0000 UTC m=+1531.212294571" watchObservedRunningTime="2026-11-03 17:59:11.756668247 +0000 UTC m=+1531.215839232"
Nov 03 17:59:12.390271 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:12.390239    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:12.391466 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:12.391443    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tlz9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:59:12.391595 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:12.391504    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:59:13.390420 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:13.390383    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tlz9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:59:13.390949 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:13.390436    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:59:14.390098 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:14.390062    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tlz9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:59:14.390282 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:14.390117    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:59:15.390484 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:15.390449    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tlz9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:59:15.390942 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:15.390506    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:59:15.893058 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:15.893028    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"]
Nov 03 17:59:15.899948 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:15.899909    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:15.907262 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:15.907227    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksq4\" (UniqueName: \"kubernetes.io/projected/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-kube-api-access-6ksq4\") pod \"payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:15.907535 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:15.907509    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:15.961725 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:15.961685    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"]
Nov 03 17:59:16.003904 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.003827    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-12a446f9\""
Nov 03 17:59:16.004090 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.003918    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-12a446f9-dockercfg-f2d5l\""
Nov 03 17:59:16.013322 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.009439    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:16.013322 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.009571    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksq4\" (UniqueName: \"kubernetes.io/projected/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-kube-api-access-6ksq4\") pod \"payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:16.013322 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.010942    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:16.075378 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.075338    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksq4\" (UniqueName: \"kubernetes.io/projected/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-kube-api-access-6ksq4\") pod \"payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:16.083037 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.083010    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"]
Nov 03 17:59:16.087650 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.087632    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.115911 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.113683    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9dhm\" (UniqueName: \"kubernetes.io/projected/d70b1805-be63-44c3-b924-ce55116bfd49-kube-api-access-l9dhm\") pod \"payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") " pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.115911 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.113969    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70b1805-be63-44c3-b924-ce55116bfd49-plugins-config-volume\") pod \"payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") " pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.162056 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.161954    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"]
Nov 03 17:59:16.228995 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.216236    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70b1805-be63-44c3-b924-ce55116bfd49-plugins-config-volume\") pod \"payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") " pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.228995 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.217184    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70b1805-be63-44c3-b924-ce55116bfd49-plugins-config-volume\") pod \"payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") " pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.228995 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.227171    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9dhm\" (UniqueName: \"kubernetes.io/projected/d70b1805-be63-44c3-b924-ce55116bfd49-kube-api-access-l9dhm\") pod \"payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") " pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.228995 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.227752    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:16.291995 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.291866    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9dhm\" (UniqueName: \"kubernetes.io/projected/d70b1805-be63-44c3-b924-ce55116bfd49-kube-api-access-l9dhm\") pod \"payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") " pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.389953 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.389911    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tlz9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:59:16.390147 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.389973    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:59:16.441974 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.423138    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:16.760992 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.760970    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"]
Nov 03 17:59:16.890286 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:16.890256    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"]
Nov 03 17:59:17.390843 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.390738    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tlz9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:59:17.390843 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.390806    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:59:17.693302 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.693200    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph" event={"ID":"d70b1805-be63-44c3-b924-ce55116bfd49","Type":"ContainerStarted","Data":"0dd4d861850ac86b193b2137fc37587888ea07d1b1c688dc26e9d918b0d1d839"}
Nov 03 17:59:17.693783 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.693318    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:17.693783 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.693346    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph" event={"ID":"d70b1805-be63-44c3-b924-ce55116bfd49","Type":"ContainerStarted","Data":"dadeb59bbfc792e5fc4aff60720e121b67a2b772383af36bf56c8a7ce0680d19"}
Nov 03 17:59:17.695115 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.695082    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp" event={"ID":"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2","Type":"ContainerStarted","Data":"b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117"}
Nov 03 17:59:17.695269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.695139    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp" event={"ID":"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2","Type":"ContainerStarted","Data":"e1fa7a756ad97586534ad82098ab64cb944bf055c08440d4af0623effb8c2700"}
Nov 03 17:59:17.695269 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.695237    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:17.818458 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:17.818407    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph" podStartSLOduration=2.818393214 podStartE2EDuration="2.818393214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:17.814770878 +0000 UTC m=+1537.273941878" watchObservedRunningTime="2026-11-03 17:59:17.818393214 +0000 UTC m=+1537.277564199"
Nov 03 17:59:18.390022 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:18.389984    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tlz9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 17:59:18.390201 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:18.390056    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 17:59:19.443970 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:19.443939    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:19.444497 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:19.444480    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:19.445324 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:19.445296    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 17:59:19.619832 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:19.619783    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp" podStartSLOduration=4.619768865 podStartE2EDuration="4.619768865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:17.948207313 +0000 UTC m=+1537.407378301" watchObservedRunningTime="2026-11-03 17:59:19.619768865 +0000 UTC m=+1539.078939851"
Nov 03 17:59:28.714466 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:28.714440    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 17:59:28.715126 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:28.714498    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 17:59:36.997659 ip-10-0-131-172 kubenswrapper[4493]: E1103 17:59:36.997615    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 17:59:40.875893 ip-10-0-131-172 kubenswrapper[4493]: I1103 17:59:40.866020    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:00:00.168590 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.168495    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"]
Nov 03 18:00:00.173442 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.173412    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.175176 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.175155    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"]
Nov 03 18:00:00.179484 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.179463    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.180523 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.180497    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"]
Nov 03 18:00:00.184769 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.184748    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.218776 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.218746    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"]
Nov 03 18:00:00.219807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.219783    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd343dc-df19-4520-bea5-fd50be8fdfbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.219960 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.219831    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vqv\" (UniqueName: \"kubernetes.io/projected/5a1a89d2-6301-44e1-add7-60080c859ae2-kube-api-access-p5vqv\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.220033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.219956    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a89d2-6301-44e1-add7-60080c859ae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.220076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.220033    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdzx\" (UniqueName: \"kubernetes.io/projected/9bd343dc-df19-4520-bea5-fd50be8fdfbf-kube-api-access-ptdzx\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.220181 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.220165    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmc2\" (UniqueName: \"kubernetes.io/projected/cc92cb53-5ce2-43fd-9c84-0614bc839393-kube-api-access-npmc2\") pod \"maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.220257 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.220241    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc92cb53-5ce2-43fd-9c84-0614bc839393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.224139 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.224121    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.225181 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.225160    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"]
Nov 03 18:00:00.230250 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.230231    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.232793 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.232773    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"]
Nov 03 18:00:00.237058 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.237040    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.240574 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.240557    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"]
Nov 03 18:00:00.245169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.245152    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.246514 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.246490    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"]
Nov 03 18:00:00.276656 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.276620    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:00:00.276793 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.276724    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:00:00.276897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.276803    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:00:00.284647 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.284403    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"]
Nov 03 18:00:00.291504 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.291478    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"]
Nov 03 18:00:00.303706 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.303682    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"]
Nov 03 18:00:00.311255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.311229    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"]
Nov 03 18:00:00.321775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.321746    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5dx\" (UniqueName: \"kubernetes.io/projected/a1370960-0d11-46fc-991d-73796362572c-kube-api-access-8c5dx\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.321931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.321797    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a89d2-6301-44e1-add7-60080c859ae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.321931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.321822    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9ph\" (UniqueName: \"kubernetes.io/projected/0a9b3253-a5d4-4625-84fc-cd62086a4770-kube-api-access-ds9ph\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.322064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.321938    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdzx\" (UniqueName: \"kubernetes.io/projected/9bd343dc-df19-4520-bea5-fd50be8fdfbf-kube-api-access-ptdzx\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.322064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.321998    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ab845-5fd1-4d09-9268-332b1a0cbc1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.322170 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322130    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmc2\" (UniqueName: \"kubernetes.io/projected/cc92cb53-5ce2-43fd-9c84-0614bc839393-kube-api-access-npmc2\") pod \"maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.322230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322166    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjs2q\" (UniqueName: \"kubernetes.io/projected/1da02f02-63c9-459c-8886-644a921b2092-kube-api-access-vjs2q\") pod \"maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.322286 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322229    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc92cb53-5ce2-43fd-9c84-0614bc839393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.322286 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322255    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b3253-a5d4-4625-84fc-cd62086a4770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.322390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322294    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da02f02-63c9-459c-8886-644a921b2092-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.322390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322329    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1370960-0d11-46fc-991d-73796362572c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.322502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322389    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmljr\" (UniqueName: \"kubernetes.io/projected/884ab845-5fd1-4d09-9268-332b1a0cbc1b-kube-api-access-rmljr\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.322502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322402    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a89d2-6301-44e1-add7-60080c859ae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.322952 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322928    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd343dc-df19-4520-bea5-fd50be8fdfbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.323065 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.322992    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vqv\" (UniqueName: \"kubernetes.io/projected/5a1a89d2-6301-44e1-add7-60080c859ae2-kube-api-access-p5vqv\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.323334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.323310    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc92cb53-5ce2-43fd-9c84-0614bc839393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.323543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.323525    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd343dc-df19-4520-bea5-fd50be8fdfbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.348963 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.348933    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"]
Nov 03 18:00:00.355610 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.355581    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"]
Nov 03 18:00:00.387026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.386995    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmc2\" (UniqueName: \"kubernetes.io/projected/cc92cb53-5ce2-43fd-9c84-0614bc839393-kube-api-access-npmc2\") pod \"maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.387186 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.387029    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdzx\" (UniqueName: \"kubernetes.io/projected/9bd343dc-df19-4520-bea5-fd50be8fdfbf-kube-api-access-ptdzx\") pod \"maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.388451 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.388424    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vqv\" (UniqueName: \"kubernetes.io/projected/5a1a89d2-6301-44e1-add7-60080c859ae2-kube-api-access-p5vqv\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.424925 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.424824    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjs2q\" (UniqueName: \"kubernetes.io/projected/1da02f02-63c9-459c-8886-644a921b2092-kube-api-access-vjs2q\") pod \"maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.425052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.424930    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b3253-a5d4-4625-84fc-cd62086a4770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.425052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.424971    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da02f02-63c9-459c-8886-644a921b2092-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.425052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425005    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1370960-0d11-46fc-991d-73796362572c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.425275 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425056    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmljr\" (UniqueName: \"kubernetes.io/projected/884ab845-5fd1-4d09-9268-332b1a0cbc1b-kube-api-access-rmljr\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.425275 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425183    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5dx\" (UniqueName: \"kubernetes.io/projected/a1370960-0d11-46fc-991d-73796362572c-kube-api-access-8c5dx\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.425275 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425222    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9ph\" (UniqueName: \"kubernetes.io/projected/0a9b3253-a5d4-4625-84fc-cd62086a4770-kube-api-access-ds9ph\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.425275 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425266    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ab845-5fd1-4d09-9268-332b1a0cbc1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.425741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425718    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b3253-a5d4-4625-84fc-cd62086a4770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.425830 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425765    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da02f02-63c9-459c-8886-644a921b2092-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.425830 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425778    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1370960-0d11-46fc-991d-73796362572c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.425951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.425931    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ab845-5fd1-4d09-9268-332b1a0cbc1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.489552 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.489516    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9ph\" (UniqueName: \"kubernetes.io/projected/0a9b3253-a5d4-4625-84fc-cd62086a4770-kube-api-access-ds9ph\") pod \"maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.489717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.489576    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjs2q\" (UniqueName: \"kubernetes.io/projected/1da02f02-63c9-459c-8886-644a921b2092-kube-api-access-vjs2q\") pod \"maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.489717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.489602    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5dx\" (UniqueName: \"kubernetes.io/projected/a1370960-0d11-46fc-991d-73796362572c-kube-api-access-8c5dx\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.489717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.489644    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmljr\" (UniqueName: \"kubernetes.io/projected/884ab845-5fd1-4d09-9268-332b1a0cbc1b-kube-api-access-rmljr\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.490429 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.490405    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:00.498857 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.498835    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:00.505054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.505030    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:00.542768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.542729    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:00.554626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.554597    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:00.580185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.579711    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:00.587304 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.587268    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:00.883048 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.883017    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"]
Nov 03 18:00:00.885026 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:00:00.884996    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1a89d2_6301_44e1_add7_60080c859ae2.slice/crio-104a3e99eee94ff6c6c040a03b997c6be55624235db9ea9621bb1bc113556f3b WatchSource:0}: Error finding container 104a3e99eee94ff6c6c040a03b997c6be55624235db9ea9621bb1bc113556f3b: Status 404 returned error can't find the container with id 104a3e99eee94ff6c6c040a03b997c6be55624235db9ea9621bb1bc113556f3b
Nov 03 18:00:00.943740 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:00:00.943699    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc92cb53_5ce2_43fd_9c84_0614bc839393.slice/crio-fbf9e5cf2ec00f5db30b97103c3c1f5ba300af8c51b9ece188ed9655d9bf27b0 WatchSource:0}: Error finding container fbf9e5cf2ec00f5db30b97103c3c1f5ba300af8c51b9ece188ed9655d9bf27b0: Status 404 returned error can't find the container with id fbf9e5cf2ec00f5db30b97103c3c1f5ba300af8c51b9ece188ed9655d9bf27b0
Nov 03 18:00:00.944357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:00.944335    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"]
Nov 03 18:00:01.028695 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:01.028669    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"]
Nov 03 18:00:01.040052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:01.040025    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"]
Nov 03 18:00:01.068723 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:01.068702    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"]
Nov 03 18:00:01.073677 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:01.073654    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"]
Nov 03 18:00:01.129692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:01.129660    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"]
Nov 03 18:00:01.130509 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:01.130481    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" event={"ID":"cc92cb53-5ce2-43fd-9c84-0614bc839393","Type":"ContainerStarted","Data":"fbf9e5cf2ec00f5db30b97103c3c1f5ba300af8c51b9ece188ed9655d9bf27b0"}
Nov 03 18:00:01.132055 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:01.132030    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" event={"ID":"5a1a89d2-6301-44e1-add7-60080c859ae2","Type":"ContainerStarted","Data":"104a3e99eee94ff6c6c040a03b997c6be55624235db9ea9621bb1bc113556f3b"}
Nov 03 18:00:01.133227 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:00:01.133207    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da02f02_63c9_459c_8886_644a921b2092.slice/crio-181d5266186ff0e1c1d91793327173c3edab9616f6cbb910668cba3f3632556f WatchSource:0}: Error finding container 181d5266186ff0e1c1d91793327173c3edab9616f6cbb910668cba3f3632556f: Status 404 returned error can't find the container with id 181d5266186ff0e1c1d91793327173c3edab9616f6cbb910668cba3f3632556f
Nov 03 18:00:01.133610 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:00:01.133589    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd343dc_df19_4520_bea5_fd50be8fdfbf.slice/crio-9b3350a93fe9ac7dfcffa45b84037de442bc74700ae788eae19b3cdd3a3908d2 WatchSource:0}: Error finding container 9b3350a93fe9ac7dfcffa45b84037de442bc74700ae788eae19b3cdd3a3908d2: Status 404 returned error can't find the container with id 9b3350a93fe9ac7dfcffa45b84037de442bc74700ae788eae19b3cdd3a3908d2
Nov 03 18:00:02.141321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.141287    4493 generic.go:356] "Generic (PLEG): container finished" podID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerID="188c9d21611b003a6cfd7c5b55af6bb8b84efa9d3f09a9dcb9e5296526882677" exitCode=60
Nov 03 18:00:02.141752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.141384    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" event={"ID":"cc92cb53-5ce2-43fd-9c84-0614bc839393","Type":"ContainerDied","Data":"188c9d21611b003a6cfd7c5b55af6bb8b84efa9d3f09a9dcb9e5296526882677"}
Nov 03 18:00:02.141813 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.141783    4493 scope.go:122] "RemoveContainer" containerID="188c9d21611b003a6cfd7c5b55af6bb8b84efa9d3f09a9dcb9e5296526882677"
Nov 03 18:00:02.143206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.143171    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerStarted","Data":"c1f090d67e5a453ccd018d1eb4b0aacb1223375d39d9eaa9026e33f803208448"}
Nov 03 18:00:02.143341 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.143236    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerStarted","Data":"4a86b722c37f4ccd7af067c5d2731028c4c2bec2eb0ce3ea8a6c55f7fb4784ec"}
Nov 03 18:00:02.144927 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.144906    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerStarted","Data":"a3b527fae92cb784d152ddda4e1740c3249ec70bb9983847dad52f183fc25ba9"}
Nov 03 18:00:02.145029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.144949    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerStarted","Data":"0735f33b345aa1885185688fca8bba2cee3df0114ba7537cfe6b2a3d304b5e04"}
Nov 03 18:00:02.146646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.146622    4493 generic.go:356] "Generic (PLEG): container finished" podID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerID="3ebf08be3e83b453fb2c995eb179b8b6553f1102a2411afd9018c0220ce8cf94" exitCode=60
Nov 03 18:00:02.146746 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.146717    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" event={"ID":"5a1a89d2-6301-44e1-add7-60080c859ae2","Type":"ContainerDied","Data":"3ebf08be3e83b453fb2c995eb179b8b6553f1102a2411afd9018c0220ce8cf94"}
Nov 03 18:00:02.147000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.146984    4493 scope.go:122] "RemoveContainer" containerID="3ebf08be3e83b453fb2c995eb179b8b6553f1102a2411afd9018c0220ce8cf94"
Nov 03 18:00:02.148659 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.148573    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerStarted","Data":"e07ee057a2440dce8994e149a08a46e071eff07e6ce821c6b9c318fa0bbe74ca"}
Nov 03 18:00:02.148659 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.148624    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerStarted","Data":"181d5266186ff0e1c1d91793327173c3edab9616f6cbb910668cba3f3632556f"}
Nov 03 18:00:02.150299 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.150278    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerStarted","Data":"e5ce61ea33f74186b27b5287b99f71738b638b04596d64abeade26f7723b4cdb"}
Nov 03 18:00:02.150412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.150324    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerStarted","Data":"9b3350a93fe9ac7dfcffa45b84037de442bc74700ae788eae19b3cdd3a3908d2"}
Nov 03 18:00:02.151933 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.151908    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerStarted","Data":"f82791da619552d29d6a30d89f2c049d1fff7cb699d7ea854dc808a47f02c4aa"}
Nov 03 18:00:02.152014 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.151956    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerStarted","Data":"16f84e51021c9d6eb8a39d9f35636d5879a418a4926ede0a1f72691110bee4ec"}
Nov 03 18:00:02.390216 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.390191    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"]
Nov 03 18:00:02.395131 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.395113    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.450700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.450656    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:02.454458 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.454433    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"]
Nov 03 18:00:02.462099 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.462068    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.462293 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.462258    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.462447 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.462424    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/05fca4ba-20d6-4444-ae3b-6ff51d61a125-kube-api-access-w9b2n\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.511335 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.511290    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" podStartSLOduration=2.511276668 podStartE2EDuration="2.511276668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.506220205 +0000 UTC m=+1581.965391228" watchObservedRunningTime="2026-11-03 18:00:02.511276668 +0000 UTC m=+1581.970447648"
Nov 03 18:00:02.563975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.563932    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.564129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.564037    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.564129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.564110    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/05fca4ba-20d6-4444-ae3b-6ff51d61a125-kube-api-access-w9b2n\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.564776 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.564702    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.566539 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.566490    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.634914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.633145    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/05fca4ba-20d6-4444-ae3b-6ff51d61a125-kube-api-access-w9b2n\") pod \"maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.634914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.634498    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:02.644030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.643979    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" podStartSLOduration=2.643961473 podStartE2EDuration="2.643961473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.638494499 +0000 UTC m=+1582.097665485" watchObservedRunningTime="2026-11-03 18:00:02.643961473 +0000 UTC m=+1582.103132459"
Nov 03 18:00:02.778772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.778714    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" podStartSLOduration=2.778696309 podStartE2EDuration="2.778696309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.777057788 +0000 UTC m=+1582.236228785" watchObservedRunningTime="2026-11-03 18:00:02.778696309 +0000 UTC m=+1582.237867296"
Nov 03 18:00:02.899994 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:02.899936    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" podStartSLOduration=2.899922538 podStartE2EDuration="2.899922538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.899601116 +0000 UTC m=+1582.358772114" watchObservedRunningTime="2026-11-03 18:00:02.899922538 +0000 UTC m=+1582.359093526"
Nov 03 18:00:03.027569 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:00:03.027541    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fca4ba_20d6_4444_ae3b_6ff51d61a125.slice/crio-9879120009bf777deb0dcce42007b5f4b602dddd786f05f650e64433425bc87a WatchSource:0}: Error finding container 9879120009bf777deb0dcce42007b5f4b602dddd786f05f650e64433425bc87a: Status 404 returned error can't find the container with id 9879120009bf777deb0dcce42007b5f4b602dddd786f05f650e64433425bc87a
Nov 03 18:00:03.030476 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.030453    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"]
Nov 03 18:00:03.033105 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.033069    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" podStartSLOduration=3.033056632 podStartE2EDuration="3.033056632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:03.024594459 +0000 UTC m=+1582.483765446" watchObservedRunningTime="2026-11-03 18:00:03.033056632 +0000 UTC m=+1582.492227644"
Nov 03 18:00:03.162550 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.162458    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" event={"ID":"cc92cb53-5ce2-43fd-9c84-0614bc839393","Type":"ContainerStarted","Data":"fbea0eb1b7a80ea8e075186a434a33c39a7ba9b61f575ee53f11cf3b28bdc554"}
Nov 03 18:00:03.164339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.164308    4493 generic.go:356] "Generic (PLEG): container finished" podID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerID="c1f090d67e5a453ccd018d1eb4b0aacb1223375d39d9eaa9026e33f803208448" exitCode=60
Nov 03 18:00:03.164429 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.164390    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerDied","Data":"c1f090d67e5a453ccd018d1eb4b0aacb1223375d39d9eaa9026e33f803208448"}
Nov 03 18:00:03.164669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.164652    4493 scope.go:122] "RemoveContainer" containerID="c1f090d67e5a453ccd018d1eb4b0aacb1223375d39d9eaa9026e33f803208448"
Nov 03 18:00:03.166287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.166264    4493 generic.go:356] "Generic (PLEG): container finished" podID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerID="a3b527fae92cb784d152ddda4e1740c3249ec70bb9983847dad52f183fc25ba9" exitCode=60
Nov 03 18:00:03.166381 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.166345    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerDied","Data":"a3b527fae92cb784d152ddda4e1740c3249ec70bb9983847dad52f183fc25ba9"}
Nov 03 18:00:03.166651 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.166585    4493 scope.go:122] "RemoveContainer" containerID="a3b527fae92cb784d152ddda4e1740c3249ec70bb9983847dad52f183fc25ba9"
Nov 03 18:00:03.168663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.168645    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" event={"ID":"5a1a89d2-6301-44e1-add7-60080c859ae2","Type":"ContainerStarted","Data":"7d20cf8257ca909e31a0c0757630765eb961e0d9be4dfb5f2fd02c8e3806b51a"}
Nov 03 18:00:03.170192 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.170148    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4" event={"ID":"05fca4ba-20d6-4444-ae3b-6ff51d61a125","Type":"ContainerStarted","Data":"9879120009bf777deb0dcce42007b5f4b602dddd786f05f650e64433425bc87a"}
Nov 03 18:00:03.171815 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.171792    4493 generic.go:356] "Generic (PLEG): container finished" podID="1da02f02-63c9-459c-8886-644a921b2092" containerID="e07ee057a2440dce8994e149a08a46e071eff07e6ce821c6b9c318fa0bbe74ca" exitCode=60
Nov 03 18:00:03.171929 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.171828    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerDied","Data":"e07ee057a2440dce8994e149a08a46e071eff07e6ce821c6b9c318fa0bbe74ca"}
Nov 03 18:00:03.172109 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.172088    4493 scope.go:122] "RemoveContainer" containerID="e07ee057a2440dce8994e149a08a46e071eff07e6ce821c6b9c318fa0bbe74ca"
Nov 03 18:00:03.173580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.173561    4493 generic.go:356] "Generic (PLEG): container finished" podID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerID="e5ce61ea33f74186b27b5287b99f71738b638b04596d64abeade26f7723b4cdb" exitCode=60
Nov 03 18:00:03.173660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.173583    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerDied","Data":"e5ce61ea33f74186b27b5287b99f71738b638b04596d64abeade26f7723b4cdb"}
Nov 03 18:00:03.173851 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.173834    4493 scope.go:122] "RemoveContainer" containerID="e5ce61ea33f74186b27b5287b99f71738b638b04596d64abeade26f7723b4cdb"
Nov 03 18:00:03.175211 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.175195    4493 generic.go:356] "Generic (PLEG): container finished" podID="a1370960-0d11-46fc-991d-73796362572c" containerID="f82791da619552d29d6a30d89f2c049d1fff7cb699d7ea854dc808a47f02c4aa" exitCode=60
Nov 03 18:00:03.175294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.175217    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerDied","Data":"f82791da619552d29d6a30d89f2c049d1fff7cb699d7ea854dc808a47f02c4aa"}
Nov 03 18:00:03.175422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.175409    4493 scope.go:122] "RemoveContainer" containerID="f82791da619552d29d6a30d89f2c049d1fff7cb699d7ea854dc808a47f02c4aa"
Nov 03 18:00:03.287584 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.287536    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" podStartSLOduration=3.287523611 podStartE2EDuration="3.287523611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:03.286080559 +0000 UTC m=+1582.745251546" watchObservedRunningTime="2026-11-03 18:00:03.287523611 +0000 UTC m=+1582.746694596"
Nov 03 18:00:03.406720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:03.406676    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" podStartSLOduration=3.406663554 podStartE2EDuration="3.406663554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:03.406428222 +0000 UTC m=+1582.865599207" watchObservedRunningTime="2026-11-03 18:00:03.406663554 +0000 UTC m=+1582.865834544"
Nov 03 18:00:04.184953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.184908    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerStarted","Data":"a1c8ab1ab6395c7e8df2862143044ff874be5b8b59d1e1d4283b154965cfbe8b"}
Nov 03 18:00:04.186980 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.186948    4493 generic.go:356] "Generic (PLEG): container finished" podID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerID="fbea0eb1b7a80ea8e075186a434a33c39a7ba9b61f575ee53f11cf3b28bdc554" exitCode=60
Nov 03 18:00:04.187117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.187012    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" event={"ID":"cc92cb53-5ce2-43fd-9c84-0614bc839393","Type":"ContainerDied","Data":"fbea0eb1b7a80ea8e075186a434a33c39a7ba9b61f575ee53f11cf3b28bdc554"}
Nov 03 18:00:04.187117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.187066    4493 scope.go:122] "RemoveContainer" containerID="188c9d21611b003a6cfd7c5b55af6bb8b84efa9d3f09a9dcb9e5296526882677"
Nov 03 18:00:04.187324 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.187302    4493 scope.go:122] "RemoveContainer" containerID="fbea0eb1b7a80ea8e075186a434a33c39a7ba9b61f575ee53f11cf3b28bdc554"
Nov 03 18:00:04.187655 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:04.187590    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4_odh-ai-gateway-infra(cc92cb53-5ce2-43fd-9c84-0614bc839393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393"
Nov 03 18:00:04.189210 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.189189    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerStarted","Data":"2734e0e23e88326182a4e65a82c0467cc520687939ce4415a0e18e4317fd279b"}
Nov 03 18:00:04.191167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.191144    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerStarted","Data":"ab7752207109398fa4cf7126b719829ceb29b6f63a02d6be4d1c0d0297cb6fb4"}
Nov 03 18:00:04.193318 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.193298    4493 generic.go:356] "Generic (PLEG): container finished" podID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerID="7d20cf8257ca909e31a0c0757630765eb961e0d9be4dfb5f2fd02c8e3806b51a" exitCode=60
Nov 03 18:00:04.193415 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.193384    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" event={"ID":"5a1a89d2-6301-44e1-add7-60080c859ae2","Type":"ContainerDied","Data":"7d20cf8257ca909e31a0c0757630765eb961e0d9be4dfb5f2fd02c8e3806b51a"}
Nov 03 18:00:04.193572 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.193557    4493 scope.go:122] "RemoveContainer" containerID="7d20cf8257ca909e31a0c0757630765eb961e0d9be4dfb5f2fd02c8e3806b51a"
Nov 03 18:00:04.193827 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:04.193810    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp_odh-ai-gateway-infra(5a1a89d2-6301-44e1-add7-60080c859ae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2"
Nov 03 18:00:04.195069 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.195049    4493 generic.go:356] "Generic (PLEG): container finished" podID="05fca4ba-20d6-4444-ae3b-6ff51d61a125" containerID="16823a8bb198fff790efb5ef2c4d56cd5647c6b3cd12705c2a4e1f7a371764ad" exitCode=0
Nov 03 18:00:04.195154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.195095    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4" event={"ID":"05fca4ba-20d6-4444-ae3b-6ff51d61a125","Type":"ContainerDied","Data":"16823a8bb198fff790efb5ef2c4d56cd5647c6b3cd12705c2a4e1f7a371764ad"}
Nov 03 18:00:04.197121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.197097    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerStarted","Data":"5811d5baa2c785b91ef4b2d9f82ea45ad11eabb1e88b8e37116103d3c2f0067a"}
Nov 03 18:00:04.199058 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.199037    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerStarted","Data":"5a7d61fef471e5d57e0a1c50028adb54ebc173ad4ce8ca2e60740632ffebb243"}
Nov 03 18:00:04.202107 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:04.202082    4493 scope.go:122] "RemoveContainer" containerID="3ebf08be3e83b453fb2c995eb179b8b6553f1102a2411afd9018c0220ce8cf94"
Nov 03 18:00:05.212101 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.212070    4493 scope.go:122] "RemoveContainer" containerID="fbea0eb1b7a80ea8e075186a434a33c39a7ba9b61f575ee53f11cf3b28bdc554"
Nov 03 18:00:05.212572 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:05.212473    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4_odh-ai-gateway-infra(cc92cb53-5ce2-43fd-9c84-0614bc839393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393"
Nov 03 18:00:05.214151 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.214121    4493 generic.go:356] "Generic (PLEG): container finished" podID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerID="2734e0e23e88326182a4e65a82c0467cc520687939ce4415a0e18e4317fd279b" exitCode=60
Nov 03 18:00:05.214329 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.214308    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerDied","Data":"2734e0e23e88326182a4e65a82c0467cc520687939ce4415a0e18e4317fd279b"}
Nov 03 18:00:05.214421 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.214366    4493 scope.go:122] "RemoveContainer" containerID="c1f090d67e5a453ccd018d1eb4b0aacb1223375d39d9eaa9026e33f803208448"
Nov 03 18:00:05.214659 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.214627    4493 scope.go:122] "RemoveContainer" containerID="2734e0e23e88326182a4e65a82c0467cc520687939ce4415a0e18e4317fd279b"
Nov 03 18:00:05.215141 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:05.215115    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd_odh-ai-gateway-infra(884ab845-5fd1-4d09-9268-332b1a0cbc1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b"
Nov 03 18:00:05.219422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.219370    4493 generic.go:356] "Generic (PLEG): container finished" podID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerID="ab7752207109398fa4cf7126b719829ceb29b6f63a02d6be4d1c0d0297cb6fb4" exitCode=60
Nov 03 18:00:05.219618 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.219457    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerDied","Data":"ab7752207109398fa4cf7126b719829ceb29b6f63a02d6be4d1c0d0297cb6fb4"}
Nov 03 18:00:05.219896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.219771    4493 scope.go:122] "RemoveContainer" containerID="ab7752207109398fa4cf7126b719829ceb29b6f63a02d6be4d1c0d0297cb6fb4"
Nov 03 18:00:05.220270 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:05.220242    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf_odh-ai-gateway-infra(0a9b3253-a5d4-4625-84fc-cd62086a4770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770"
Nov 03 18:00:05.222370 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.222346    4493 scope.go:122] "RemoveContainer" containerID="7d20cf8257ca909e31a0c0757630765eb961e0d9be4dfb5f2fd02c8e3806b51a"
Nov 03 18:00:05.222761 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:05.222737    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp_odh-ai-gateway-infra(5a1a89d2-6301-44e1-add7-60080c859ae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2"
Nov 03 18:00:05.224596 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.224575    4493 generic.go:356] "Generic (PLEG): container finished" podID="1da02f02-63c9-459c-8886-644a921b2092" containerID="5811d5baa2c785b91ef4b2d9f82ea45ad11eabb1e88b8e37116103d3c2f0067a" exitCode=60
Nov 03 18:00:05.224708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.224688    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerDied","Data":"5811d5baa2c785b91ef4b2d9f82ea45ad11eabb1e88b8e37116103d3c2f0067a"}
Nov 03 18:00:05.225024 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.225003    4493 scope.go:122] "RemoveContainer" containerID="5811d5baa2c785b91ef4b2d9f82ea45ad11eabb1e88b8e37116103d3c2f0067a"
Nov 03 18:00:05.225574 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:05.225551    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8_odh-ai-gateway-infra(1da02f02-63c9-459c-8886-644a921b2092)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" podUID="1da02f02-63c9-459c-8886-644a921b2092"
Nov 03 18:00:05.227902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.227838    4493 generic.go:356] "Generic (PLEG): container finished" podID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerID="5a7d61fef471e5d57e0a1c50028adb54ebc173ad4ce8ca2e60740632ffebb243" exitCode=60
Nov 03 18:00:05.228021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.227997    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerDied","Data":"5a7d61fef471e5d57e0a1c50028adb54ebc173ad4ce8ca2e60740632ffebb243"}
Nov 03 18:00:05.228276 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.228258    4493 scope.go:122] "RemoveContainer" containerID="5a7d61fef471e5d57e0a1c50028adb54ebc173ad4ce8ca2e60740632ffebb243"
Nov 03 18:00:05.228671 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:05.228646    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6_odh-ai-gateway-infra(9bd343dc-df19-4520-bea5-fd50be8fdfbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf"
Nov 03 18:00:05.230311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.230287    4493 generic.go:356] "Generic (PLEG): container finished" podID="a1370960-0d11-46fc-991d-73796362572c" containerID="a1c8ab1ab6395c7e8df2862143044ff874be5b8b59d1e1d4283b154965cfbe8b" exitCode=60
Nov 03 18:00:05.230425 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.230374    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerDied","Data":"a1c8ab1ab6395c7e8df2862143044ff874be5b8b59d1e1d4283b154965cfbe8b"}
Nov 03 18:00:05.230646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.230631    4493 scope.go:122] "RemoveContainer" containerID="a1c8ab1ab6395c7e8df2862143044ff874be5b8b59d1e1d4283b154965cfbe8b"
Nov 03 18:00:05.230988 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:05.230969    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p_odh-ai-gateway-infra(a1370960-0d11-46fc-991d-73796362572c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" podUID="a1370960-0d11-46fc-991d-73796362572c"
Nov 03 18:00:05.232546 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.232517    4493 scope.go:122] "RemoveContainer" containerID="a3b527fae92cb784d152ddda4e1740c3249ec70bb9983847dad52f183fc25ba9"
Nov 03 18:00:05.252973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.252947    4493 scope.go:122] "RemoveContainer" containerID="e07ee057a2440dce8994e149a08a46e071eff07e6ce821c6b9c318fa0bbe74ca"
Nov 03 18:00:05.274043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.274018    4493 scope.go:122] "RemoveContainer" containerID="e5ce61ea33f74186b27b5287b99f71738b638b04596d64abeade26f7723b4cdb"
Nov 03 18:00:05.290503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.290476    4493 scope.go:122] "RemoveContainer" containerID="f82791da619552d29d6a30d89f2c049d1fff7cb699d7ea854dc808a47f02c4aa"
Nov 03 18:00:05.435657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.435618    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:05.507691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.507661    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fca4ba-20d6-4444-ae3b-6ff51d61a125-kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/05fca4ba-20d6-4444-ae3b-6ff51d61a125-kube-api-access-w9b2n\") pod \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") "
Nov 03 18:00:05.507859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.507832    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-xks-ca\") pod \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") "
Nov 03 18:00:05.507974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.507953    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-openshift-ca\") pod \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\" (UID: \"05fca4ba-20d6-4444-ae3b-6ff51d61a125\") "
Nov 03 18:00:05.508312 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.508288    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-openshift-ca" pod "05fca4ba-20d6-4444-ae3b-6ff51d61a125" (UID: "05fca4ba-20d6-4444-ae3b-6ff51d61a125"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:05.508553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.508500    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:05.509929 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.509905    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fca4ba-20d6-4444-ae3b-6ff51d61a125-kube-api-access-w9b2n" pod "05fca4ba-20d6-4444-ae3b-6ff51d61a125" (UID: "05fca4ba-20d6-4444-ae3b-6ff51d61a125"). InnerVolumeSpecName "kube-api-access-w9b2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:05.510108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.510094    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-xks-ca" pod "05fca4ba-20d6-4444-ae3b-6ff51d61a125" (UID: "05fca4ba-20d6-4444-ae3b-6ff51d61a125"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:05.610394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.610357    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9b2n\" (UniqueName: \"kubernetes.io/projected/05fca4ba-20d6-4444-ae3b-6ff51d61a125-kube-api-access-w9b2n\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:05.610394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:05.610389    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05fca4ba-20d6-4444-ae3b-6ff51d61a125-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:06.246969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.246940    4493 scope.go:122] "RemoveContainer" containerID="2734e0e23e88326182a4e65a82c0467cc520687939ce4415a0e18e4317fd279b"
Nov 03 18:00:06.247466 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:06.247366    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd_odh-ai-gateway-infra(884ab845-5fd1-4d09-9268-332b1a0cbc1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b"
Nov 03 18:00:06.249138 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.249107    4493 scope.go:122] "RemoveContainer" containerID="ab7752207109398fa4cf7126b719829ceb29b6f63a02d6be4d1c0d0297cb6fb4"
Nov 03 18:00:06.249492 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:06.249453    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf_odh-ai-gateway-infra(0a9b3253-a5d4-4625-84fc-cd62086a4770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770"
Nov 03 18:00:06.250742 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.250725    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"
Nov 03 18:00:06.250852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.250786    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4" event={"ID":"05fca4ba-20d6-4444-ae3b-6ff51d61a125","Type":"ContainerDied","Data":"9879120009bf777deb0dcce42007b5f4b602dddd786f05f650e64433425bc87a"}
Nov 03 18:00:06.250852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.250836    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9879120009bf777deb0dcce42007b5f4b602dddd786f05f650e64433425bc87a"
Nov 03 18:00:06.253253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.253235    4493 scope.go:122] "RemoveContainer" containerID="5811d5baa2c785b91ef4b2d9f82ea45ad11eabb1e88b8e37116103d3c2f0067a"
Nov 03 18:00:06.253547 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:06.253508    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8_odh-ai-gateway-infra(1da02f02-63c9-459c-8886-644a921b2092)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" podUID="1da02f02-63c9-459c-8886-644a921b2092"
Nov 03 18:00:06.255132 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.255113    4493 scope.go:122] "RemoveContainer" containerID="5a7d61fef471e5d57e0a1c50028adb54ebc173ad4ce8ca2e60740632ffebb243"
Nov 03 18:00:06.255430 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:06.255405    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6_odh-ai-gateway-infra(9bd343dc-df19-4520-bea5-fd50be8fdfbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf"
Nov 03 18:00:06.256834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:06.256818    4493 scope.go:122] "RemoveContainer" containerID="a1c8ab1ab6395c7e8df2862143044ff874be5b8b59d1e1d4283b154965cfbe8b"
Nov 03 18:00:06.257066 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:06.257048    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p_odh-ai-gateway-infra(a1370960-0d11-46fc-991d-73796362572c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" podUID="a1370960-0d11-46fc-991d-73796362572c"
Nov 03 18:00:06.994270 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:06.994231    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:00:13.706290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:13.706257    4493 scope.go:122] "RemoveContainer" containerID="7d20cf8257ca909e31a0c0757630765eb961e0d9be4dfb5f2fd02c8e3806b51a"
Nov 03 18:00:14.343465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:14.343377    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" event={"ID":"5a1a89d2-6301-44e1-add7-60080c859ae2","Type":"ContainerStarted","Data":"8ca54e30b20d0f00e39f710a9522f1f7ba149052a1d8725e487e4770d1e71a96"}
Nov 03 18:00:15.353530 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:15.353494    4493 generic.go:356] "Generic (PLEG): container finished" podID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerID="8ca54e30b20d0f00e39f710a9522f1f7ba149052a1d8725e487e4770d1e71a96" exitCode=60
Nov 03 18:00:15.354037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:15.353554    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" event={"ID":"5a1a89d2-6301-44e1-add7-60080c859ae2","Type":"ContainerDied","Data":"8ca54e30b20d0f00e39f710a9522f1f7ba149052a1d8725e487e4770d1e71a96"}
Nov 03 18:00:15.354037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:15.353601    4493 scope.go:122] "RemoveContainer" containerID="7d20cf8257ca909e31a0c0757630765eb961e0d9be4dfb5f2fd02c8e3806b51a"
Nov 03 18:00:15.354155 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:15.354066    4493 scope.go:122] "RemoveContainer" containerID="8ca54e30b20d0f00e39f710a9522f1f7ba149052a1d8725e487e4770d1e71a96"
Nov 03 18:00:15.354503 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:15.354470    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp_odh-ai-gateway-infra(5a1a89d2-6301-44e1-add7-60080c859ae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2"
Nov 03 18:00:15.490489 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:15.490455    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"]
Nov 03 18:00:15.706247 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:15.706163    4493 scope.go:122] "RemoveContainer" containerID="5811d5baa2c785b91ef4b2d9f82ea45ad11eabb1e88b8e37116103d3c2f0067a"
Nov 03 18:00:15.706494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:15.706477    4493 scope.go:122] "RemoveContainer" containerID="fbea0eb1b7a80ea8e075186a434a33c39a7ba9b61f575ee53f11cf3b28bdc554"
Nov 03 18:00:16.048240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.048204    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"]
Nov 03 18:00:16.049183 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.049166    4493 cpu_manager.go:414] "containerMap: removing container" podUID="05fca4ba-20d6-4444-ae3b-6ff51d61a125" containerName="revoke-keys"
Nov 03 18:00:16.049230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.049187    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fca4ba-20d6-4444-ae3b-6ff51d61a125" containerName="revoke-keys"
Nov 03 18:00:16.049379 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.049369    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fca4ba-20d6-4444-ae3b-6ff51d61a125" containerName="revoke-keys"
Nov 03 18:00:16.053277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.053260    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.111746 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.111716    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"]
Nov 03 18:00:16.111984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.111971    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:16.123746 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.123719    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.124222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.124195    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.124338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.124233    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhdb\" (UniqueName: \"kubernetes.io/projected/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-kube-api-access-xfhdb\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.225626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.225599    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.225820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.225717    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.225820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.225737    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhdb\" (UniqueName: \"kubernetes.io/projected/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-kube-api-access-xfhdb\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.226394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.226373    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.228115 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.228090    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.290022 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.289961    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhdb\" (UniqueName: \"kubernetes.io/projected/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-kube-api-access-xfhdb\") pod \"maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.363945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.363906    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" event={"ID":"cc92cb53-5ce2-43fd-9c84-0614bc839393","Type":"ContainerStarted","Data":"56957a4801e02599db87e82e88810f0d87cb6926180a676d4de463d21c92110a"}
Nov 03 18:00:16.366271 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.366174    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:16.367921 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.367895    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerStarted","Data":"f1ca462c2ae55787821e2f46312763820b2b3d7fb17c1e54dbf9f07ebd38d48f"}
Nov 03 18:00:16.686104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.686023    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:16.787469 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:00:16.787325    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bc0271_4426_4ba4_9fe7_cabc6804fdc7.slice/crio-009eb2450c6220febbf05f28b0cd94c965fd37fc4add05a5bb3f8cbcf881275d WatchSource:0}: Error finding container 009eb2450c6220febbf05f28b0cd94c965fd37fc4add05a5bb3f8cbcf881275d: Status 404 returned error can't find the container with id 009eb2450c6220febbf05f28b0cd94c965fd37fc4add05a5bb3f8cbcf881275d
Nov 03 18:00:16.789156 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.789134    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"]
Nov 03 18:00:16.892000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.885455    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1a89d2-6301-44e1-add7-60080c859ae2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a89d2-6301-44e1-add7-60080c859ae2-maas-api-service-ca\") pod \"5a1a89d2-6301-44e1-add7-60080c859ae2\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") "
Nov 03 18:00:16.892000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.885612    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1a89d2-6301-44e1-add7-60080c859ae2-kube-api-access-p5vqv\" (UniqueName: \"kubernetes.io/projected/5a1a89d2-6301-44e1-add7-60080c859ae2-kube-api-access-p5vqv\") pod \"5a1a89d2-6301-44e1-add7-60080c859ae2\" (UID: \"5a1a89d2-6301-44e1-add7-60080c859ae2\") "
Nov 03 18:00:16.910101 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.907418    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1a89d2-6301-44e1-add7-60080c859ae2-kube-api-access-p5vqv" pod "5a1a89d2-6301-44e1-add7-60080c859ae2" (UID: "5a1a89d2-6301-44e1-add7-60080c859ae2"). InnerVolumeSpecName "kube-api-access-p5vqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:16.910408 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.910363    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1a89d2-6301-44e1-add7-60080c859ae2-maas-api-service-ca" pod "5a1a89d2-6301-44e1-add7-60080c859ae2" (UID: "5a1a89d2-6301-44e1-add7-60080c859ae2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:16.987839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.987809    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a1a89d2-6301-44e1-add7-60080c859ae2-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:16.987839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:16.987838    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vqv\" (UniqueName: \"kubernetes.io/projected/5a1a89d2-6301-44e1-add7-60080c859ae2-kube-api-access-p5vqv\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:17.377832 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.377790    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd" event={"ID":"46bc0271-4426-4ba4-9fe7-cabc6804fdc7","Type":"ContainerStarted","Data":"2f4fde627bb9ed96031d5ea09c77ad9559d301eda64e484e424e36e71e3a2cfa"}
Nov 03 18:00:17.378330 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.377931    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd" event={"ID":"46bc0271-4426-4ba4-9fe7-cabc6804fdc7","Type":"ContainerStarted","Data":"009eb2450c6220febbf05f28b0cd94c965fd37fc4add05a5bb3f8cbcf881275d"}
Nov 03 18:00:17.379912 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.379855    4493 generic.go:356] "Generic (PLEG): container finished" podID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerID="56957a4801e02599db87e82e88810f0d87cb6926180a676d4de463d21c92110a" exitCode=60
Nov 03 18:00:17.380056 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.379911    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" event={"ID":"cc92cb53-5ce2-43fd-9c84-0614bc839393","Type":"ContainerDied","Data":"56957a4801e02599db87e82e88810f0d87cb6926180a676d4de463d21c92110a"}
Nov 03 18:00:17.380056 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.379966    4493 scope.go:122] "RemoveContainer" containerID="fbea0eb1b7a80ea8e075186a434a33c39a7ba9b61f575ee53f11cf3b28bdc554"
Nov 03 18:00:17.380469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.380441    4493 scope.go:122] "RemoveContainer" containerID="56957a4801e02599db87e82e88810f0d87cb6926180a676d4de463d21c92110a"
Nov 03 18:00:17.380863 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:17.380834    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4_odh-ai-gateway-infra(cc92cb53-5ce2-43fd-9c84-0614bc839393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393"
Nov 03 18:00:17.381595 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.381579    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"
Nov 03 18:00:17.381667 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.381607    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp" event={"ID":"5a1a89d2-6301-44e1-add7-60080c859ae2","Type":"ContainerDied","Data":"104a3e99eee94ff6c6c040a03b997c6be55624235db9ea9621bb1bc113556f3b"}
Nov 03 18:00:17.383800 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.383780    4493 generic.go:356] "Generic (PLEG): container finished" podID="1da02f02-63c9-459c-8886-644a921b2092" containerID="f1ca462c2ae55787821e2f46312763820b2b3d7fb17c1e54dbf9f07ebd38d48f" exitCode=60
Nov 03 18:00:17.383917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.383843    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerDied","Data":"f1ca462c2ae55787821e2f46312763820b2b3d7fb17c1e54dbf9f07ebd38d48f"}
Nov 03 18:00:17.384267 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.384248    4493 scope.go:122] "RemoveContainer" containerID="f1ca462c2ae55787821e2f46312763820b2b3d7fb17c1e54dbf9f07ebd38d48f"
Nov 03 18:00:17.384538 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:17.384521    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8_odh-ai-gateway-infra(1da02f02-63c9-459c-8886-644a921b2092)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" podUID="1da02f02-63c9-459c-8886-644a921b2092"
Nov 03 18:00:17.394464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.394445    4493 scope.go:122] "RemoveContainer" containerID="8ca54e30b20d0f00e39f710a9522f1f7ba149052a1d8725e487e4770d1e71a96"
Nov 03 18:00:17.410063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.410037    4493 scope.go:122] "RemoveContainer" containerID="5811d5baa2c785b91ef4b2d9f82ea45ad11eabb1e88b8e37116103d3c2f0067a"
Nov 03 18:00:17.500288 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.500214    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"]
Nov 03 18:00:17.500619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.500582    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd" podStartSLOduration=2.500571529 podStartE2EDuration="2.500571529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:17.497323741 +0000 UTC m=+1596.956494721" watchObservedRunningTime="2026-11-03 18:00:17.500571529 +0000 UTC m=+1596.959742514"
Nov 03 18:00:17.705941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.705907    4493 scope.go:122] "RemoveContainer" containerID="ab7752207109398fa4cf7126b719829ceb29b6f63a02d6be4d1c0d0297cb6fb4"
Nov 03 18:00:17.706094 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.705976    4493 scope.go:122] "RemoveContainer" containerID="5a7d61fef471e5d57e0a1c50028adb54ebc173ad4ce8ca2e60740632ffebb243"
Nov 03 18:00:17.738983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.738955    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"]
Nov 03 18:00:17.747506 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.747479    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895480-25llp"]
Nov 03 18:00:17.752753 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:17.752688    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"]
Nov 03 18:00:18.394995 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.394961    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerStarted","Data":"353e036d53e90ecef3b9b23bb1a3aef66d87d2116641b2eee68077feb11656d2"}
Nov 03 18:00:18.400287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.400252    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerStarted","Data":"6c79ebd0b482b26b1198797fb087b73b23e4cac0c8f2ba78c63949821944e85a"}
Nov 03 18:00:18.402125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.402091    4493 generic.go:356] "Generic (PLEG): container finished" podID="46bc0271-4426-4ba4-9fe7-cabc6804fdc7" containerID="2f4fde627bb9ed96031d5ea09c77ad9559d301eda64e484e424e36e71e3a2cfa" exitCode=0
Nov 03 18:00:18.402236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.402180    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd" event={"ID":"46bc0271-4426-4ba4-9fe7-cabc6804fdc7","Type":"ContainerDied","Data":"2f4fde627bb9ed96031d5ea09c77ad9559d301eda64e484e424e36e71e3a2cfa"}
Nov 03 18:00:18.562302 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.562257    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:18.604640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.604618    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:18.614169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.614145    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc92cb53-5ce2-43fd-9c84-0614bc839393-kube-api-access-npmc2\" (UniqueName: \"kubernetes.io/projected/cc92cb53-5ce2-43fd-9c84-0614bc839393-kube-api-access-npmc2\") pod \"cc92cb53-5ce2-43fd-9c84-0614bc839393\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") "
Nov 03 18:00:18.614291 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.614256    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc92cb53-5ce2-43fd-9c84-0614bc839393-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc92cb53-5ce2-43fd-9c84-0614bc839393-maas-api-service-ca\") pod \"cc92cb53-5ce2-43fd-9c84-0614bc839393\" (UID: \"cc92cb53-5ce2-43fd-9c84-0614bc839393\") "
Nov 03 18:00:18.614585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.614561    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc92cb53-5ce2-43fd-9c84-0614bc839393-maas-api-service-ca" pod "cc92cb53-5ce2-43fd-9c84-0614bc839393" (UID: "cc92cb53-5ce2-43fd-9c84-0614bc839393"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:18.614959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.614941    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc92cb53-5ce2-43fd-9c84-0614bc839393-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:18.616397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.616374    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc92cb53-5ce2-43fd-9c84-0614bc839393-kube-api-access-npmc2" pod "cc92cb53-5ce2-43fd-9c84-0614bc839393" (UID: "cc92cb53-5ce2-43fd-9c84-0614bc839393"). InnerVolumeSpecName "kube-api-access-npmc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:18.706991 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.706911    4493 scope.go:122] "RemoveContainer" containerID="2734e0e23e88326182a4e65a82c0467cc520687939ce4415a0e18e4317fd279b"
Nov 03 18:00:18.707167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.707073    4493 scope.go:122] "RemoveContainer" containerID="a1c8ab1ab6395c7e8df2862143044ff874be5b8b59d1e1d4283b154965cfbe8b"
Nov 03 18:00:18.715839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.715817    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da02f02-63c9-459c-8886-644a921b2092-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da02f02-63c9-459c-8886-644a921b2092-maas-api-service-ca\") pod \"1da02f02-63c9-459c-8886-644a921b2092\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") "
Nov 03 18:00:18.715952 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.715858    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da02f02-63c9-459c-8886-644a921b2092-kube-api-access-vjs2q\" (UniqueName: \"kubernetes.io/projected/1da02f02-63c9-459c-8886-644a921b2092-kube-api-access-vjs2q\") pod \"1da02f02-63c9-459c-8886-644a921b2092\" (UID: \"1da02f02-63c9-459c-8886-644a921b2092\") "
Nov 03 18:00:18.716220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.716190    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da02f02-63c9-459c-8886-644a921b2092-maas-api-service-ca" pod "1da02f02-63c9-459c-8886-644a921b2092" (UID: "1da02f02-63c9-459c-8886-644a921b2092"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:18.716346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.716322    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" path="/var/lib/kubelet/pods/5a1a89d2-6301-44e1-add7-60080c859ae2/volumes"
Nov 03 18:00:18.716555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.716536    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da02f02-63c9-459c-8886-644a921b2092-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:18.716613 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.716562    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmc2\" (UniqueName: \"kubernetes.io/projected/cc92cb53-5ce2-43fd-9c84-0614bc839393-kube-api-access-npmc2\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:18.719760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.719733    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da02f02-63c9-459c-8886-644a921b2092-kube-api-access-vjs2q" pod "1da02f02-63c9-459c-8886-644a921b2092" (UID: "1da02f02-63c9-459c-8886-644a921b2092"). InnerVolumeSpecName "kube-api-access-vjs2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:18.818784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:18.818691    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjs2q\" (UniqueName: \"kubernetes.io/projected/1da02f02-63c9-459c-8886-644a921b2092-kube-api-access-vjs2q\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:19.412436 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.412398    4493 generic.go:356] "Generic (PLEG): container finished" podID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerID="353e036d53e90ecef3b9b23bb1a3aef66d87d2116641b2eee68077feb11656d2" exitCode=60
Nov 03 18:00:19.412916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.412467    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerDied","Data":"353e036d53e90ecef3b9b23bb1a3aef66d87d2116641b2eee68077feb11656d2"}
Nov 03 18:00:19.412916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.412548    4493 scope.go:122] "RemoveContainer" containerID="ab7752207109398fa4cf7126b719829ceb29b6f63a02d6be4d1c0d0297cb6fb4"
Nov 03 18:00:19.413086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.413070    4493 scope.go:122] "RemoveContainer" containerID="353e036d53e90ecef3b9b23bb1a3aef66d87d2116641b2eee68077feb11656d2"
Nov 03 18:00:19.413815 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:19.413495    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf_odh-ai-gateway-infra(0a9b3253-a5d4-4625-84fc-cd62086a4770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770"
Nov 03 18:00:19.415361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.414330    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"
Nov 03 18:00:19.415361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.414332    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8" event={"ID":"1da02f02-63c9-459c-8886-644a921b2092","Type":"ContainerDied","Data":"181d5266186ff0e1c1d91793327173c3edab9616f6cbb910668cba3f3632556f"}
Nov 03 18:00:19.420547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.420521    4493 generic.go:356] "Generic (PLEG): container finished" podID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerID="6c79ebd0b482b26b1198797fb087b73b23e4cac0c8f2ba78c63949821944e85a" exitCode=60
Nov 03 18:00:19.420671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.420640    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerDied","Data":"6c79ebd0b482b26b1198797fb087b73b23e4cac0c8f2ba78c63949821944e85a"}
Nov 03 18:00:19.421098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.421060    4493 scope.go:122] "RemoveContainer" containerID="6c79ebd0b482b26b1198797fb087b73b23e4cac0c8f2ba78c63949821944e85a"
Nov 03 18:00:19.421515 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:19.421487    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6_odh-ai-gateway-infra(9bd343dc-df19-4520-bea5-fd50be8fdfbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf"
Nov 03 18:00:19.424352 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.424329    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerStarted","Data":"c45aea24b756102a1b21af88d2e2d491e9145bab3fdbe62407f7be7689699fe1"}
Nov 03 18:00:19.426244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.426222    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4" event={"ID":"cc92cb53-5ce2-43fd-9c84-0614bc839393","Type":"ContainerDied","Data":"fbf9e5cf2ec00f5db30b97103c3c1f5ba300af8c51b9ece188ed9655d9bf27b0"}
Nov 03 18:00:19.426340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.426326    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"
Nov 03 18:00:19.429564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.429543    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerStarted","Data":"5e9f897f56422f632086e28a6c4ee6cfbbe747328473d3d9503a6c387035c3d0"}
Nov 03 18:00:19.432693 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.432675    4493 scope.go:122] "RemoveContainer" containerID="f1ca462c2ae55787821e2f46312763820b2b3d7fb17c1e54dbf9f07ebd38d48f"
Nov 03 18:00:19.445798 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.445779    4493 scope.go:122] "RemoveContainer" containerID="5a7d61fef471e5d57e0a1c50028adb54ebc173ad4ce8ca2e60740632ffebb243"
Nov 03 18:00:19.462759 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.462729    4493 scope.go:122] "RemoveContainer" containerID="56957a4801e02599db87e82e88810f0d87cb6926180a676d4de463d21c92110a"
Nov 03 18:00:19.614232 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.614204    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:19.656033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.655997    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"]
Nov 03 18:00:19.661118 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.661081    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-07f297-29895480-szjz4"]
Nov 03 18:00:19.736223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.736193    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-openshift-ca\") pod \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") "
Nov 03 18:00:19.736407 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.736382    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-kube-api-access-xfhdb\" (UniqueName: \"kubernetes.io/projected/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-kube-api-access-xfhdb\") pod \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") "
Nov 03 18:00:19.736477 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.736458    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-xks-ca\") pod \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\" (UID: \"46bc0271-4426-4ba4-9fe7-cabc6804fdc7\") "
Nov 03 18:00:19.736563 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.736539    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-openshift-ca" pod "46bc0271-4426-4ba4-9fe7-cabc6804fdc7" (UID: "46bc0271-4426-4ba4-9fe7-cabc6804fdc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:19.737226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.737131    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:19.738642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.738616    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-kube-api-access-xfhdb" pod "46bc0271-4426-4ba4-9fe7-cabc6804fdc7" (UID: "46bc0271-4426-4ba4-9fe7-cabc6804fdc7"). InnerVolumeSpecName "kube-api-access-xfhdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:19.738719 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.738658    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-xks-ca" pod "46bc0271-4426-4ba4-9fe7-cabc6804fdc7" (UID: "46bc0271-4426-4ba4-9fe7-cabc6804fdc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:19.777290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.777239    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"]
Nov 03 18:00:19.783240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.783210    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"]
Nov 03 18:00:19.790257 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.790226    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1c1ea78c-29895480-qvzn8"]
Nov 03 18:00:19.838570 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.838484    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhdb\" (UniqueName: \"kubernetes.io/projected/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-kube-api-access-xfhdb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:19.838570 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.838516    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc0271-4426-4ba4-9fe7-cabc6804fdc7-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:19.899778 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:19.899745    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"]
Nov 03 18:00:20.440143 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.440113    4493 generic.go:356] "Generic (PLEG): container finished" podID="a1370960-0d11-46fc-991d-73796362572c" containerID="c45aea24b756102a1b21af88d2e2d491e9145bab3fdbe62407f7be7689699fe1" exitCode=60
Nov 03 18:00:20.440625 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.440191    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerDied","Data":"c45aea24b756102a1b21af88d2e2d491e9145bab3fdbe62407f7be7689699fe1"}
Nov 03 18:00:20.440625 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.440268    4493 scope.go:122] "RemoveContainer" containerID="a1c8ab1ab6395c7e8df2862143044ff874be5b8b59d1e1d4283b154965cfbe8b"
Nov 03 18:00:20.440742 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.440638    4493 scope.go:122] "RemoveContainer" containerID="c45aea24b756102a1b21af88d2e2d491e9145bab3fdbe62407f7be7689699fe1"
Nov 03 18:00:20.441066 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:20.441041    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p_odh-ai-gateway-infra(a1370960-0d11-46fc-991d-73796362572c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" podUID="a1370960-0d11-46fc-991d-73796362572c"
Nov 03 18:00:20.441728 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.441710    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"
Nov 03 18:00:20.441818 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.441744    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd" event={"ID":"46bc0271-4426-4ba4-9fe7-cabc6804fdc7","Type":"ContainerDied","Data":"009eb2450c6220febbf05f28b0cd94c965fd37fc4add05a5bb3f8cbcf881275d"}
Nov 03 18:00:20.441818 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.441789    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="009eb2450c6220febbf05f28b0cd94c965fd37fc4add05a5bb3f8cbcf881275d"
Nov 03 18:00:20.444794 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.444770    4493 generic.go:356] "Generic (PLEG): container finished" podID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerID="5e9f897f56422f632086e28a6c4ee6cfbbe747328473d3d9503a6c387035c3d0" exitCode=60
Nov 03 18:00:20.444921 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.444846    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerDied","Data":"5e9f897f56422f632086e28a6c4ee6cfbbe747328473d3d9503a6c387035c3d0"}
Nov 03 18:00:20.445260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.445243    4493 scope.go:122] "RemoveContainer" containerID="5e9f897f56422f632086e28a6c4ee6cfbbe747328473d3d9503a6c387035c3d0"
Nov 03 18:00:20.445600 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:20.445572    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd_odh-ai-gateway-infra(884ab845-5fd1-4d09-9268-332b1a0cbc1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b"
Nov 03 18:00:20.456838 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.456819    4493 scope.go:122] "RemoveContainer" containerID="2734e0e23e88326182a4e65a82c0467cc520687939ce4415a0e18e4317fd279b"
Nov 03 18:00:20.665521 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.665493    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:20.668480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.668454    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:20.714743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.714712    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da02f02-63c9-459c-8886-644a921b2092" path="/var/lib/kubelet/pods/1da02f02-63c9-459c-8886-644a921b2092/volumes"
Nov 03 18:00:20.715191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.715175    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" path="/var/lib/kubelet/pods/cc92cb53-5ce2-43fd-9c84-0614bc839393/volumes"
Nov 03 18:00:20.751027 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.751000    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9b3253-a5d4-4625-84fc-cd62086a4770-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b3253-a5d4-4625-84fc-cd62086a4770-maas-api-service-ca\") pod \"0a9b3253-a5d4-4625-84fc-cd62086a4770\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") "
Nov 03 18:00:20.751197 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.751184    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd343dc-df19-4520-bea5-fd50be8fdfbf-kube-api-access-ptdzx\" (UniqueName: \"kubernetes.io/projected/9bd343dc-df19-4520-bea5-fd50be8fdfbf-kube-api-access-ptdzx\") pod \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") "
Nov 03 18:00:20.751306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.751294    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9b3253-a5d4-4625-84fc-cd62086a4770-kube-api-access-ds9ph\" (UniqueName: \"kubernetes.io/projected/0a9b3253-a5d4-4625-84fc-cd62086a4770-kube-api-access-ds9ph\") pod \"0a9b3253-a5d4-4625-84fc-cd62086a4770\" (UID: \"0a9b3253-a5d4-4625-84fc-cd62086a4770\") "
Nov 03 18:00:20.751343 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.751319    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9b3253-a5d4-4625-84fc-cd62086a4770-maas-api-service-ca" pod "0a9b3253-a5d4-4625-84fc-cd62086a4770" (UID: "0a9b3253-a5d4-4625-84fc-cd62086a4770"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:20.751392 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.751358    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd343dc-df19-4520-bea5-fd50be8fdfbf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd343dc-df19-4520-bea5-fd50be8fdfbf-maas-api-service-ca\") pod \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\" (UID: \"9bd343dc-df19-4520-bea5-fd50be8fdfbf\") "
Nov 03 18:00:20.751785 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.751757    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd343dc-df19-4520-bea5-fd50be8fdfbf-maas-api-service-ca" pod "9bd343dc-df19-4520-bea5-fd50be8fdfbf" (UID: "9bd343dc-df19-4520-bea5-fd50be8fdfbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:20.752220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.752202    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd343dc-df19-4520-bea5-fd50be8fdfbf-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:20.752220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.752219    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9b3253-a5d4-4625-84fc-cd62086a4770-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:20.753547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.753530    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd343dc-df19-4520-bea5-fd50be8fdfbf-kube-api-access-ptdzx" pod "9bd343dc-df19-4520-bea5-fd50be8fdfbf" (UID: "9bd343dc-df19-4520-bea5-fd50be8fdfbf"). InnerVolumeSpecName "kube-api-access-ptdzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:20.753644 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.753625    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9b3253-a5d4-4625-84fc-cd62086a4770-kube-api-access-ds9ph" pod "0a9b3253-a5d4-4625-84fc-cd62086a4770" (UID: "0a9b3253-a5d4-4625-84fc-cd62086a4770"). InnerVolumeSpecName "kube-api-access-ds9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:20.854216 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.854178    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptdzx\" (UniqueName: \"kubernetes.io/projected/9bd343dc-df19-4520-bea5-fd50be8fdfbf-kube-api-access-ptdzx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:20.854216 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:20.854219    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9ph\" (UniqueName: \"kubernetes.io/projected/0a9b3253-a5d4-4625-84fc-cd62086a4770-kube-api-access-ds9ph\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.055620 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.055540    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"]
Nov 03 18:00:21.181054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.181017    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"]
Nov 03 18:00:21.461317 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.461295    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"
Nov 03 18:00:21.461710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.461291    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf" event={"ID":"0a9b3253-a5d4-4625-84fc-cd62086a4770","Type":"ContainerDied","Data":"0735f33b345aa1885185688fca8bba2cee3df0114ba7537cfe6b2a3d304b5e04"}
Nov 03 18:00:21.461710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.461435    4493 scope.go:122] "RemoveContainer" containerID="353e036d53e90ecef3b9b23bb1a3aef66d87d2116641b2eee68077feb11656d2"
Nov 03 18:00:21.463462 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.463435    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"
Nov 03 18:00:21.463566 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.463472    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6" event={"ID":"9bd343dc-df19-4520-bea5-fd50be8fdfbf","Type":"ContainerDied","Data":"9b3350a93fe9ac7dfcffa45b84037de442bc74700ae788eae19b3cdd3a3908d2"}
Nov 03 18:00:21.475193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.475155    4493 scope.go:122] "RemoveContainer" containerID="6c79ebd0b482b26b1198797fb087b73b23e4cac0c8f2ba78c63949821944e85a"
Nov 03 18:00:21.638503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.638479    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:21.684201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.684179    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:21.704626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.704598    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"]
Nov 03 18:00:21.713865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.713844    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-f7d463-29895480-fcqwf"]
Nov 03 18:00:21.766934 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.766906    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884ab845-5fd1-4d09-9268-332b1a0cbc1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ab845-5fd1-4d09-9268-332b1a0cbc1b-maas-api-service-ca\") pod \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") "
Nov 03 18:00:21.767103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.766944    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1370960-0d11-46fc-991d-73796362572c-kube-api-access-8c5dx\" (UniqueName: \"kubernetes.io/projected/a1370960-0d11-46fc-991d-73796362572c-kube-api-access-8c5dx\") pod \"a1370960-0d11-46fc-991d-73796362572c\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") "
Nov 03 18:00:21.767103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.766982    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1370960-0d11-46fc-991d-73796362572c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1370960-0d11-46fc-991d-73796362572c-maas-api-service-ca\") pod \"a1370960-0d11-46fc-991d-73796362572c\" (UID: \"a1370960-0d11-46fc-991d-73796362572c\") "
Nov 03 18:00:21.767103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.767040    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884ab845-5fd1-4d09-9268-332b1a0cbc1b-kube-api-access-rmljr\" (UniqueName: \"kubernetes.io/projected/884ab845-5fd1-4d09-9268-332b1a0cbc1b-kube-api-access-rmljr\") pod \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\" (UID: \"884ab845-5fd1-4d09-9268-332b1a0cbc1b\") "
Nov 03 18:00:21.767339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.767315    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884ab845-5fd1-4d09-9268-332b1a0cbc1b-maas-api-service-ca" pod "884ab845-5fd1-4d09-9268-332b1a0cbc1b" (UID: "884ab845-5fd1-4d09-9268-332b1a0cbc1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:21.767396 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.767336    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1370960-0d11-46fc-991d-73796362572c-maas-api-service-ca" pod "a1370960-0d11-46fc-991d-73796362572c" (UID: "a1370960-0d11-46fc-991d-73796362572c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:21.768081 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.768056    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884ab845-5fd1-4d09-9268-332b1a0cbc1b-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.768081 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.768082    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1370960-0d11-46fc-991d-73796362572c-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.769401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.769377    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1370960-0d11-46fc-991d-73796362572c-kube-api-access-8c5dx" pod "a1370960-0d11-46fc-991d-73796362572c" (UID: "a1370960-0d11-46fc-991d-73796362572c"). InnerVolumeSpecName "kube-api-access-8c5dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:21.769521 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.769503    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884ab845-5fd1-4d09-9268-332b1a0cbc1b-kube-api-access-rmljr" pod "884ab845-5fd1-4d09-9268-332b1a0cbc1b" (UID: "884ab845-5fd1-4d09-9268-332b1a0cbc1b"). InnerVolumeSpecName "kube-api-access-rmljr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:21.826785 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.826738    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"]
Nov 03 18:00:21.832373 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.832346    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-42c162-29895480-w9zl6"]
Nov 03 18:00:21.869193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.869166    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5dx\" (UniqueName: \"kubernetes.io/projected/a1370960-0d11-46fc-991d-73796362572c-kube-api-access-8c5dx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:21.869193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:21.869188    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmljr\" (UniqueName: \"kubernetes.io/projected/884ab845-5fd1-4d09-9268-332b1a0cbc1b-kube-api-access-rmljr\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:22.482601 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.482566    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"
Nov 03 18:00:22.483115 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.482586    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd" event={"ID":"884ab845-5fd1-4d09-9268-332b1a0cbc1b","Type":"ContainerDied","Data":"4a86b722c37f4ccd7af067c5d2731028c4c2bec2eb0ce3ea8a6c55f7fb4784ec"}
Nov 03 18:00:22.483115 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.482701    4493 scope.go:122] "RemoveContainer" containerID="5e9f897f56422f632086e28a6c4ee6cfbbe747328473d3d9503a6c387035c3d0"
Nov 03 18:00:22.492987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.492951    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p" event={"ID":"a1370960-0d11-46fc-991d-73796362572c","Type":"ContainerDied","Data":"16f84e51021c9d6eb8a39d9f35636d5879a418a4926ede0a1f72691110bee4ec"}
Nov 03 18:00:22.493129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.493009    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"
Nov 03 18:00:22.499979 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.499945    4493 scope.go:122] "RemoveContainer" containerID="c45aea24b756102a1b21af88d2e2d491e9145bab3fdbe62407f7be7689699fe1"
Nov 03 18:00:22.658563 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.658525    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"]
Nov 03 18:00:22.664268 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.664243    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895480-kscsd"]
Nov 03 18:00:22.714480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.714451    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" path="/var/lib/kubelet/pods/0a9b3253-a5d4-4625-84fc-cd62086a4770/volumes"
Nov 03 18:00:22.715054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.715029    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" path="/var/lib/kubelet/pods/884ab845-5fd1-4d09-9268-332b1a0cbc1b/volumes"
Nov 03 18:00:22.715585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.715567    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" path="/var/lib/kubelet/pods/9bd343dc-df19-4520-bea5-fd50be8fdfbf/volumes"
Nov 03 18:00:22.775152 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.775080    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"]
Nov 03 18:00:22.784083 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:22.784054    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895480-b294p"]
Nov 03 18:00:24.714903 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:24.714844    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1370960-0d11-46fc-991d-73796362572c" path="/var/lib/kubelet/pods/a1370960-0d11-46fc-991d-73796362572c/volumes"
Nov 03 18:00:29.072425 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:29.072388    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"]
Nov 03 18:00:29.124703 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:29.124653    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0831a2b-hd8dd"]
Nov 03 18:00:30.719857 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:30.719818    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3201103-f5be-48f9-95fa-e61ed2602ad5" path="/var/lib/kubelet/pods/f3201103-f5be-48f9-95fa-e61ed2602ad5/volumes"
Nov 03 18:00:36.993087 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:00:36.993048    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:00:44.121465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.121436    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"]
Nov 03 18:00:44.122109 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122092    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122111    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122122    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122127    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122134    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122139    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122149    4493 cpu_manager.go:414] "containerMap: removing container" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122154    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122160    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:00:44.122161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122164    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122172    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122177    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122187    4493 cpu_manager.go:414] "containerMap: removing container" podUID="46bc0271-4426-4ba4-9fe7-cabc6804fdc7" containerName="revoke-keys"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122191    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bc0271-4426-4ba4-9fe7-cabc6804fdc7" containerName="revoke-keys"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122205    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122209    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122218    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122222    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122231    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122235    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122241    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122245    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122251    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122255    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122261    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122264    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122274    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122278    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122287    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122291    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122299    4493 cpu_manager.go:414] "containerMap: removing container" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122303    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122310    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122313    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122398    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122406    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122411    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122417    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122423    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122429    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122437    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122442    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122448    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122453    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:00:44.122444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122458    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:00:44.123532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122464    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bc0271-4426-4ba4-9fe7-cabc6804fdc7" containerName="revoke-keys"
Nov 03 18:00:44.123532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122472    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.123532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122476    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:00:44.123532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122481    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:00:44.123532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122486    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.123532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.122490    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da02f02-63c9-459c-8886-644a921b2092" containerName="cleanup"
Nov 03 18:00:44.127575 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.127558    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.184254 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.184220    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"]
Nov 03 18:00:44.205623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.205562    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n54n\" (UniqueName: \"kubernetes.io/projected/474ba459-96df-422a-9421-7328a57e6db4-kube-api-access-5n54n\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.205825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.205752    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474ba459-96df-422a-9421-7328a57e6db4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.205936 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.205852    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474ba459-96df-422a-9421-7328a57e6db4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.229137 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.229089    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:00:44.229137 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.229117    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:00:44.229347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.229168    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:00:44.232134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.232109    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:00:44.307300 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.307271    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474ba459-96df-422a-9421-7328a57e6db4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.307478 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.307359    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474ba459-96df-422a-9421-7328a57e6db4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.307558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.307542    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n54n\" (UniqueName: \"kubernetes.io/projected/474ba459-96df-422a-9421-7328a57e6db4-kube-api-access-5n54n\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.308145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.308120    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474ba459-96df-422a-9421-7328a57e6db4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.309917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.309860    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474ba459-96df-422a-9421-7328a57e6db4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.371536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.371465    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n54n\" (UniqueName: \"kubernetes.io/projected/474ba459-96df-422a-9421-7328a57e6db4-kube-api-access-5n54n\") pod \"maas-api-revoke-keys-e2e-derive-aa500779-mvp7j\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.441333 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.441304    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:44.825222 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:00:44.825190    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474ba459_96df_422a_9421_7328a57e6db4.slice/crio-5578ca063eb6ef91a70adc5acdf173a9d5226a0fc90d086d3abcabd3c31124d0 WatchSource:0}: Error finding container 5578ca063eb6ef91a70adc5acdf173a9d5226a0fc90d086d3abcabd3c31124d0: Status 404 returned error can't find the container with id 5578ca063eb6ef91a70adc5acdf173a9d5226a0fc90d086d3abcabd3c31124d0
Nov 03 18:00:44.826976 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:44.826952    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"]
Nov 03 18:00:45.695598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:45.695503    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j" event={"ID":"474ba459-96df-422a-9421-7328a57e6db4","Type":"ContainerStarted","Data":"644120ed8180276c82cbe2fc035d3acc4d9d96efbfb5e5f157c6ea7a54673d4c"}
Nov 03 18:00:45.695598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:45.695560    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j" event={"ID":"474ba459-96df-422a-9421-7328a57e6db4","Type":"ContainerStarted","Data":"5578ca063eb6ef91a70adc5acdf173a9d5226a0fc90d086d3abcabd3c31124d0"}
Nov 03 18:00:45.868306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:45.868248    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j" podStartSLOduration=1.86823458 podStartE2EDuration="1.86823458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:45.817162067 +0000 UTC m=+1625.276333053" watchObservedRunningTime="2026-11-03 18:00:45.86823458 +0000 UTC m=+1625.327405610"
Nov 03 18:00:46.705049 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:46.705016    4493 generic.go:356] "Generic (PLEG): container finished" podID="474ba459-96df-422a-9421-7328a57e6db4" containerID="644120ed8180276c82cbe2fc035d3acc4d9d96efbfb5e5f157c6ea7a54673d4c" exitCode=0
Nov 03 18:00:46.713103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:46.713074    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j" event={"ID":"474ba459-96df-422a-9421-7328a57e6db4","Type":"ContainerDied","Data":"644120ed8180276c82cbe2fc035d3acc4d9d96efbfb5e5f157c6ea7a54673d4c"}
Nov 03 18:00:47.879617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:47.879582    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:48.059905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.059791    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474ba459-96df-422a-9421-7328a57e6db4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474ba459-96df-422a-9421-7328a57e6db4-maas-api-xks-ca\") pod \"474ba459-96df-422a-9421-7328a57e6db4\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") "
Nov 03 18:00:48.060082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.060035    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/474ba459-96df-422a-9421-7328a57e6db4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474ba459-96df-422a-9421-7328a57e6db4-maas-api-openshift-ca\") pod \"474ba459-96df-422a-9421-7328a57e6db4\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") "
Nov 03 18:00:48.060252 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.060227    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474ba459-96df-422a-9421-7328a57e6db4-kube-api-access-5n54n\" (UniqueName: \"kubernetes.io/projected/474ba459-96df-422a-9421-7328a57e6db4-kube-api-access-5n54n\") pod \"474ba459-96df-422a-9421-7328a57e6db4\" (UID: \"474ba459-96df-422a-9421-7328a57e6db4\") "
Nov 03 18:00:48.060711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.060535    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474ba459-96df-422a-9421-7328a57e6db4-maas-api-openshift-ca" pod "474ba459-96df-422a-9421-7328a57e6db4" (UID: "474ba459-96df-422a-9421-7328a57e6db4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:00:48.061093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.060845    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/474ba459-96df-422a-9421-7328a57e6db4-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:48.065147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.065109    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474ba459-96df-422a-9421-7328a57e6db4-maas-api-xks-ca" pod "474ba459-96df-422a-9421-7328a57e6db4" (UID: "474ba459-96df-422a-9421-7328a57e6db4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:00:48.065538 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.065501    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474ba459-96df-422a-9421-7328a57e6db4-kube-api-access-5n54n" pod "474ba459-96df-422a-9421-7328a57e6db4" (UID: "474ba459-96df-422a-9421-7328a57e6db4"). InnerVolumeSpecName "kube-api-access-5n54n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:00:48.162729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.162689    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n54n\" (UniqueName: \"kubernetes.io/projected/474ba459-96df-422a-9421-7328a57e6db4-kube-api-access-5n54n\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:48.162729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.162722    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/474ba459-96df-422a-9421-7328a57e6db4-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:00:48.725632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.725607    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"
Nov 03 18:00:48.725777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.725643    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j" event={"ID":"474ba459-96df-422a-9421-7328a57e6db4","Type":"ContainerDied","Data":"5578ca063eb6ef91a70adc5acdf173a9d5226a0fc90d086d3abcabd3c31124d0"}
Nov 03 18:00:48.725777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:00:48.725696    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5578ca063eb6ef91a70adc5acdf173a9d5226a0fc90d086d3abcabd3c31124d0"
Nov 03 18:01:01.809889 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:01.809848    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"]
Nov 03 18:01:01.863861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:01.863809    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c74b2c26-xsnm4"]
Nov 03 18:01:02.713703 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:02.713670    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05fca4ba-20d6-4444-ae3b-6ff51d61a125" path="/var/lib/kubelet/pods/05fca4ba-20d6-4444-ae3b-6ff51d61a125/volumes"
Nov 03 18:01:06.994867 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:01:06.994826    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:01:09.304698 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:09.304654    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"]
Nov 03 18:01:09.357464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:09.357432    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c1ea78c-dp6bd"]
Nov 03 18:01:11.004677 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:11.004637    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bc0271-4426-4ba4-9fe7-cabc6804fdc7" path="/var/lib/kubelet/pods/46bc0271-4426-4ba4-9fe7-cabc6804fdc7/volumes"
Nov 03 18:01:17.069500 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:17.069464    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"]
Nov 03 18:01:17.123592 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:17.123531    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1dff07d-5wb6l"]
Nov 03 18:01:18.170907 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.170857    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"]
Nov 03 18:01:18.171535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171518    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:01:18.171580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171537    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:01:18.171580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171558    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:01:18.171580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171562    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc92cb53-5ce2-43fd-9c84-0614bc839393" containerName="cleanup"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171580    4493 cpu_manager.go:414] "containerMap: removing container" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171586    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171593    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171597    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171611    4493 cpu_manager.go:414] "containerMap: removing container" podUID="474ba459-96df-422a-9421-7328a57e6db4" containerName="revoke-keys"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171616    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474ba459-96df-422a-9421-7328a57e6db4" containerName="revoke-keys"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171634    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:01:18.171675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171638    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:01:18.171941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171726    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1370960-0d11-46fc-991d-73796362572c" containerName="cleanup"
Nov 03 18:01:18.171941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171738    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1a89d2-6301-44e1-add7-60080c859ae2" containerName="cleanup"
Nov 03 18:01:18.171941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171747    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="884ab845-5fd1-4d09-9268-332b1a0cbc1b" containerName="cleanup"
Nov 03 18:01:18.171941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171751    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9b3253-a5d4-4625-84fc-cd62086a4770" containerName="cleanup"
Nov 03 18:01:18.171941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171759    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="474ba459-96df-422a-9421-7328a57e6db4" containerName="revoke-keys"
Nov 03 18:01:18.171941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.171765    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd343dc-df19-4520-bea5-fd50be8fdfbf" containerName="cleanup"
Nov 03 18:01:18.176981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.176959    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.235502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.235464    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"]
Nov 03 18:01:18.263919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.263848    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.264202 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.264175    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvx62\" (UniqueName: \"kubernetes.io/projected/71c74362-a1e5-4f87-81d2-3f6560edafd1-kube-api-access-tvx62\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.264378 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.264356    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.280340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.280306    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:01:18.280340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.280306    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:01:18.280599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.280391    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:01:18.281263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.281240    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:01:18.366289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.366245    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvx62\" (UniqueName: \"kubernetes.io/projected/71c74362-a1e5-4f87-81d2-3f6560edafd1-kube-api-access-tvx62\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.366466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.366363    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.366466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.366421    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.367248 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.367205    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.368960 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.368933    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.430361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.430290    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvx62\" (UniqueName: \"kubernetes.io/projected/71c74362-a1e5-4f87-81d2-3f6560edafd1-kube-api-access-tvx62\") pod \"maas-api-revoke-sub-6c1b7f95-rtqjl\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.492194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.492158    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:18.714315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.714289    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a376f4a-0dda-4f61-9588-c9bf339184f2" path="/var/lib/kubelet/pods/2a376f4a-0dda-4f61-9588-c9bf339184f2/volumes"
Nov 03 18:01:18.881038 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:01:18.881005    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c74362_a1e5_4f87_81d2_3f6560edafd1.slice/crio-4cbf70f87b57d2081ae1f2fcc3a70fa900628570edb00c0e2dcc3e082557e129 WatchSource:0}: Error finding container 4cbf70f87b57d2081ae1f2fcc3a70fa900628570edb00c0e2dcc3e082557e129: Status 404 returned error can't find the container with id 4cbf70f87b57d2081ae1f2fcc3a70fa900628570edb00c0e2dcc3e082557e129
Nov 03 18:01:18.881909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:18.881884    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"]
Nov 03 18:01:19.107538 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:19.107449    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl" event={"ID":"71c74362-a1e5-4f87-81d2-3f6560edafd1","Type":"ContainerStarted","Data":"4cbf70f87b57d2081ae1f2fcc3a70fa900628570edb00c0e2dcc3e082557e129"}
Nov 03 18:01:20.185960 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:20.180956    4493 generic.go:356] "Generic (PLEG): container finished" podID="71c74362-a1e5-4f87-81d2-3f6560edafd1" containerID="04a66f8d0559e883f84a8d46e906599a488b618325aa11666262d5e8c251e79c" exitCode=0
Nov 03 18:01:20.185960 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:20.181225    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl" event={"ID":"71c74362-a1e5-4f87-81d2-3f6560edafd1","Type":"ContainerDied","Data":"04a66f8d0559e883f84a8d46e906599a488b618325aa11666262d5e8c251e79c"}
Nov 03 18:01:21.488860 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.488829    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:21.551610 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.551578    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c74362-a1e5-4f87-81d2-3f6560edafd1-kube-api-access-tvx62\" (UniqueName: \"kubernetes.io/projected/71c74362-a1e5-4f87-81d2-3f6560edafd1-kube-api-access-tvx62\") pod \"71c74362-a1e5-4f87-81d2-3f6560edafd1\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") "
Nov 03 18:01:21.551610 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.551635    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-openshift-ca\") pod \"71c74362-a1e5-4f87-81d2-3f6560edafd1\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") "
Nov 03 18:01:21.552178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.551841    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-xks-ca\") pod \"71c74362-a1e5-4f87-81d2-3f6560edafd1\" (UID: \"71c74362-a1e5-4f87-81d2-3f6560edafd1\") "
Nov 03 18:01:21.552248 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.552182    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-openshift-ca" pod "71c74362-a1e5-4f87-81d2-3f6560edafd1" (UID: "71c74362-a1e5-4f87-81d2-3f6560edafd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:21.552781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.552656    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:21.554346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.554313    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-xks-ca" pod "71c74362-a1e5-4f87-81d2-3f6560edafd1" (UID: "71c74362-a1e5-4f87-81d2-3f6560edafd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:21.554558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.554539    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c74362-a1e5-4f87-81d2-3f6560edafd1-kube-api-access-tvx62" pod "71c74362-a1e5-4f87-81d2-3f6560edafd1" (UID: "71c74362-a1e5-4f87-81d2-3f6560edafd1"). InnerVolumeSpecName "kube-api-access-tvx62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:21.654458 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.654403    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvx62\" (UniqueName: \"kubernetes.io/projected/71c74362-a1e5-4f87-81d2-3f6560edafd1-kube-api-access-tvx62\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:21.654458 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:21.654457    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c74362-a1e5-4f87-81d2-3f6560edafd1-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:22.204452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:22.204411    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl" event={"ID":"71c74362-a1e5-4f87-81d2-3f6560edafd1","Type":"ContainerDied","Data":"4cbf70f87b57d2081ae1f2fcc3a70fa900628570edb00c0e2dcc3e082557e129"}
Nov 03 18:01:22.204660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:22.204481    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cbf70f87b57d2081ae1f2fcc3a70fa900628570edb00c0e2dcc3e082557e129"
Nov 03 18:01:22.204660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:22.204555    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"
Nov 03 18:01:26.068950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:26.068915    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"]
Nov 03 18:01:26.124774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:26.124733    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e14f778-dllmd"]
Nov 03 18:01:26.714548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:26.714512    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf615a6f-c67c-4eef-98bd-f411b7f37fac" path="/var/lib/kubelet/pods/bf615a6f-c67c-4eef-98bd-f411b7f37fac/volumes"
Nov 03 18:01:29.158961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.158920    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"]
Nov 03 18:01:29.159639 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.159621    4493 cpu_manager.go:414] "containerMap: removing container" podUID="71c74362-a1e5-4f87-81d2-3f6560edafd1" containerName="revoke-keys"
Nov 03 18:01:29.159686 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.159641    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c74362-a1e5-4f87-81d2-3f6560edafd1" containerName="revoke-keys"
Nov 03 18:01:29.159750 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.159741    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c74362-a1e5-4f87-81d2-3f6560edafd1" containerName="revoke-keys"
Nov 03 18:01:29.163905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.163865    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.227158 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.227124    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 18:01:29.227356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.227125    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 18:01:29.227356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.227126    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 18:01:29.227356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.227311    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-96fc4\""
Nov 03 18:01:29.262532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.262496    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"]
Nov 03 18:01:29.282844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.282812    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.283086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.282903    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.283086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.282995    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.283086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.283025    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.283277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.283220    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b85984f-07ac-4d69-bbce-bd760c705376-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.283346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.283334    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.283478 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.283463    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7d5\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-kube-api-access-mg7d5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.385207 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.385167    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.385399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.385267    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7d5\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-kube-api-access-mg7d5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.385399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.385334    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.385399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.385397    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.385555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.385530    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.385592 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.385579    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.385692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.385676    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b85984f-07ac-4d69-bbce-bd760c705376-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.386325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.386292    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.387995 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.387960    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.388106 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.388006    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b85984f-07ac-4d69-bbce-bd760c705376-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.388106 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.388000    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.388106 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.388066    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.452315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.452235    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.452566 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.452536    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7d5\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-kube-api-access-mg7d5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5jzlq\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.477291 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.477258    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:29.720133 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.720103    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"]
Nov 03 18:01:29.724756 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.724734    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.780195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.780082    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:01:29.780195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.780174    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:01:29.786253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.786220    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"]
Nov 03 18:01:29.791795 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.791759    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdbv7\" (UniqueName: \"kubernetes.io/projected/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-kube-api-access-zdbv7\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.792070 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.791921    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.792653 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.792199    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.853426 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.853394    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:01:29.853603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.853581    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:01:29.894201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.894165    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.894401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.894377    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.894540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.894523    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdbv7\" (UniqueName: \"kubernetes.io/projected/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-kube-api-access-zdbv7\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.894959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.894919    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.896909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.896884    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:29.916256 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:01:29.916223    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b85984f_07ac_4d69_bbce_bd760c705376.slice/crio-a9dff895231a0f4b6c0e0c6bf39a9bf7aecb0cf2034d35071c71cf8b2af80d4b WatchSource:0}: Error finding container a9dff895231a0f4b6c0e0c6bf39a9bf7aecb0cf2034d35071c71cf8b2af80d4b: Status 404 returned error can't find the container with id a9dff895231a0f4b6c0e0c6bf39a9bf7aecb0cf2034d35071c71cf8b2af80d4b
Nov 03 18:01:29.917036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.917008    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"]
Nov 03 18:01:29.918550 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.918534    4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:01:29.958538 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:29.958506    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdbv7\" (UniqueName: \"kubernetes.io/projected/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-kube-api-access-zdbv7\") pod \"maas-api-revoke-keys-e2e-mt-12a446f9-475nw\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:30.039285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:30.039199    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:30.282323 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:30.282292    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" event={"ID":"9b85984f-07ac-4d69-bbce-bd760c705376","Type":"ContainerStarted","Data":"a9dff895231a0f4b6c0e0c6bf39a9bf7aecb0cf2034d35071c71cf8b2af80d4b"}
Nov 03 18:01:30.427029 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:01:30.426995    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3c893e_9a68_4d4d_bdfb_dbd4308a4781.slice/crio-f2bdaccc2f30da05f69f23c10fc7868982a73dd85316b67d373239579de2dd4e WatchSource:0}: Error finding container f2bdaccc2f30da05f69f23c10fc7868982a73dd85316b67d373239579de2dd4e: Status 404 returned error can't find the container with id f2bdaccc2f30da05f69f23c10fc7868982a73dd85316b67d373239579de2dd4e
Nov 03 18:01:30.428324 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:30.428301    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"]
Nov 03 18:01:31.292065 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:31.292028    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw" event={"ID":"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781","Type":"ContainerStarted","Data":"624778107ce668e83dfaf820ec60da247deda286fca5198fc92f37f76872e937"}
Nov 03 18:01:31.292452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:31.292092    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw" event={"ID":"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781","Type":"ContainerStarted","Data":"f2bdaccc2f30da05f69f23c10fc7868982a73dd85316b67d373239579de2dd4e"}
Nov 03 18:01:31.415884 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:31.415821    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw" podStartSLOduration=2.415807292 podStartE2EDuration="2.415807292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:31.412069683 +0000 UTC m=+1670.871240693" watchObservedRunningTime="2026-11-03 18:01:31.415807292 +0000 UTC m=+1670.874978277"
Nov 03 18:01:32.302302 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:32.302264    4493 generic.go:356] "Generic (PLEG): container finished" podID="0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" containerID="624778107ce668e83dfaf820ec60da247deda286fca5198fc92f37f76872e937" exitCode=0
Nov 03 18:01:32.302767 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:32.302336    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw" event={"ID":"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781","Type":"ContainerDied","Data":"624778107ce668e83dfaf820ec60da247deda286fca5198fc92f37f76872e937"}
Nov 03 18:01:33.463464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.463436    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:33.545851 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.545820    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-kube-api-access-zdbv7\" (UniqueName: \"kubernetes.io/projected/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-kube-api-access-zdbv7\") pod \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") "
Nov 03 18:01:33.546079 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.545957    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-xks-ca\") pod \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") "
Nov 03 18:01:33.546120 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.546112    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-openshift-ca\") pod \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\" (UID: \"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781\") "
Nov 03 18:01:33.546461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.546424    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-openshift-ca" pod "0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" (UID: "0d3c893e-9a68-4d4d-bdfb-dbd4308a4781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:33.546734 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.546679    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:33.548153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.548121    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-kube-api-access-zdbv7" pod "0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" (UID: "0d3c893e-9a68-4d4d-bdfb-dbd4308a4781"). InnerVolumeSpecName "kube-api-access-zdbv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:33.548272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.548242    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-xks-ca" pod "0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" (UID: "0d3c893e-9a68-4d4d-bdfb-dbd4308a4781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:33.647920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.647815    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdbv7\" (UniqueName: \"kubernetes.io/projected/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-kube-api-access-zdbv7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:33.647920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:33.647854    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:34.319077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:34.318989    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"
Nov 03 18:01:34.319077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:34.319029    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw" event={"ID":"0d3c893e-9a68-4d4d-bdfb-dbd4308a4781","Type":"ContainerDied","Data":"f2bdaccc2f30da05f69f23c10fc7868982a73dd85316b67d373239579de2dd4e"}
Nov 03 18:01:34.319280 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:34.319117    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2bdaccc2f30da05f69f23c10fc7868982a73dd85316b67d373239579de2dd4e"
Nov 03 18:01:36.992996 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:01:36.992960    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:01:37.038829 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:37.038790    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:01:37.038957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:37.038930    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:01:37.348449 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:37.348360    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" event={"ID":"9b85984f-07ac-4d69-bbce-bd760c705376","Type":"ContainerStarted","Data":"fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9"}
Nov 03 18:01:37.348635 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:37.348492    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:37.578643 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:37.578586    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" podStartSLOduration=1.460234652 podStartE2EDuration="8.57857099s" totalImagesPullingTime="7.118336338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:37.52395265 +0000 UTC m=+1676.983123633" watchObservedRunningTime="2026-11-03 18:01:37.57857099 +0000 UTC m=+1677.037741975"
Nov 03 18:01:38.364788 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:38.364746    4493 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5jzlq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:01:38.365317 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:38.364813    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" podUID="9b85984f-07ac-4d69-bbce-bd760c705376" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:01:40.067262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:40.067225    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"]
Nov 03 18:01:40.120199 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:40.120164    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a654930f-cdfft"]
Nov 03 18:01:40.715014 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:40.714984    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c0579e-477e-4a21-9e1b-ac2bd06f6164" path="/var/lib/kubelet/pods/c8c0579e-477e-4a21-9e1b-ac2bd06f6164/volumes"
Nov 03 18:01:41.078998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:41.078912    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"]
Nov 03 18:01:41.129505 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:41.129471    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feae5603-dkvl9"]
Nov 03 18:01:41.273410 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:41.273371    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"]
Nov 03 18:01:41.336289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:41.336202    4493 scope.go:122] "RemoveContainer" containerID="ceae5a62a99192cdcdf2fae8ea86758b348dae59338ef31783ecde4450ff4e4b"
Nov 03 18:01:41.360725 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:41.360690    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:01:41.377488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:41.376008    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"]
Nov 03 18:01:42.723638 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:42.723601    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff29423-9286-45fe-8b42-451d6117cdd8" path="/var/lib/kubelet/pods/2ff29423-9286-45fe-8b42-451d6117cdd8/volumes"
Nov 03 18:01:43.071535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:43.071445    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"]
Nov 03 18:01:43.108662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:43.108629    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e64633c7-67twq"]
Nov 03 18:01:44.715289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:44.715257    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385e15fc-41a0-4ccc-8129-82036ab6f8eb" path="/var/lib/kubelet/pods/385e15fc-41a0-4ccc-8129-82036ab6f8eb/volumes"
Nov 03 18:01:44.753605 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:44.753561    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"]
Nov 03 18:01:44.773826 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:44.773575    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aa500779-mvp7j"]
Nov 03 18:01:45.068147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:45.068052    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"]
Nov 03 18:01:45.074582 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:45.074552    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8551fa4-8548d"]
Nov 03 18:01:46.274211 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.274154    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph" podUID="d70b1805-be63-44c3-b924-ce55116bfd49" containerName="payload-processing" containerID="cri-o://0dd4d861850ac86b193b2137fc37587888ea07d1b1c688dc26e9d918b0d1d839" gracePeriod=25
Nov 03 18:01:46.377063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.377013    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp" podUID="f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" containerName="payload-pre-processing" containerID="cri-o://b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117" gracePeriod=25
Nov 03 18:01:46.440046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.440015    4493 generic.go:356] "Generic (PLEG): container finished" podID="d70b1805-be63-44c3-b924-ce55116bfd49" containerID="0dd4d861850ac86b193b2137fc37587888ea07d1b1c688dc26e9d918b0d1d839" exitCode=0
Nov 03 18:01:46.440226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.440086    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph" event={"ID":"d70b1805-be63-44c3-b924-ce55116bfd49","Type":"ContainerDied","Data":"0dd4d861850ac86b193b2137fc37587888ea07d1b1c688dc26e9d918b0d1d839"}
Nov 03 18:01:46.562862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.562831    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 18:01:46.672565 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.672538    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 18:01:46.690068 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.690040    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70b1805-be63-44c3-b924-ce55116bfd49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70b1805-be63-44c3-b924-ce55116bfd49-plugins-config-volume\") pod \"d70b1805-be63-44c3-b924-ce55116bfd49\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") "
Nov 03 18:01:46.690292 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.690276    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70b1805-be63-44c3-b924-ce55116bfd49-kube-api-access-l9dhm\" (UniqueName: \"kubernetes.io/projected/d70b1805-be63-44c3-b924-ce55116bfd49-kube-api-access-l9dhm\") pod \"d70b1805-be63-44c3-b924-ce55116bfd49\" (UID: \"d70b1805-be63-44c3-b924-ce55116bfd49\") "
Nov 03 18:01:46.690512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.690489    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70b1805-be63-44c3-b924-ce55116bfd49-plugins-config-volume" pod "d70b1805-be63-44c3-b924-ce55116bfd49" (UID: "d70b1805-be63-44c3-b924-ce55116bfd49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:46.690816 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.690792    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70b1805-be63-44c3-b924-ce55116bfd49-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:46.692548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.692528    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70b1805-be63-44c3-b924-ce55116bfd49-kube-api-access-l9dhm" pod "d70b1805-be63-44c3-b924-ce55116bfd49" (UID: "d70b1805-be63-44c3-b924-ce55116bfd49"). InnerVolumeSpecName "kube-api-access-l9dhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:46.714294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.714262    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474ba459-96df-422a-9421-7328a57e6db4" path="/var/lib/kubelet/pods/474ba459-96df-422a-9421-7328a57e6db4/volumes"
Nov 03 18:01:46.714779 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.714759    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3dcaed-470a-4a37-8193-95005dae91e1" path="/var/lib/kubelet/pods/8a3dcaed-470a-4a37-8193-95005dae91e1/volumes"
Nov 03 18:01:46.792497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.792410    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-kube-api-access-6ksq4\" (UniqueName: \"kubernetes.io/projected/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-kube-api-access-6ksq4\") pod \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") "
Nov 03 18:01:46.792497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.792465    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-plugins-config-volume\") pod \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\" (UID: \"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2\") "
Nov 03 18:01:46.792970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.792940    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-plugins-config-volume" pod "f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" (UID: "f3c4b715-d6e7-4167-b98c-1c3c0992f9b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:46.793252 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.793232    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:46.793338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.793259    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9dhm\" (UniqueName: \"kubernetes.io/projected/d70b1805-be63-44c3-b924-ce55116bfd49-kube-api-access-l9dhm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:46.794683 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.794662    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-kube-api-access-6ksq4" pod "f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" (UID: "f3c4b715-d6e7-4167-b98c-1c3c0992f9b2"). InnerVolumeSpecName "kube-api-access-6ksq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:46.894416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:46.894380    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ksq4\" (UniqueName: \"kubernetes.io/projected/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2-kube-api-access-6ksq4\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:47.449649 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.449617    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"
Nov 03 18:01:47.450145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.449617    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph" event={"ID":"d70b1805-be63-44c3-b924-ce55116bfd49","Type":"ContainerDied","Data":"dadeb59bbfc792e5fc4aff60720e121b67a2b772383af36bf56c8a7ce0680d19"}
Nov 03 18:01:47.450145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.449748    4493 scope.go:122] "RemoveContainer" containerID="0dd4d861850ac86b193b2137fc37587888ea07d1b1c688dc26e9d918b0d1d839"
Nov 03 18:01:47.451176 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.451152    4493 generic.go:356] "Generic (PLEG): container finished" podID="f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" containerID="b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117" exitCode=0
Nov 03 18:01:47.451257 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.451202    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"
Nov 03 18:01:47.451257 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.451230    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp" event={"ID":"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2","Type":"ContainerDied","Data":"b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117"}
Nov 03 18:01:47.451339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.451275    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp" event={"ID":"f3c4b715-d6e7-4167-b98c-1c3c0992f9b2","Type":"ContainerDied","Data":"e1fa7a756ad97586534ad82098ab64cb944bf055c08440d4af0623effb8c2700"}
Nov 03 18:01:47.461669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.461645    4493 scope.go:122] "RemoveContainer" containerID="b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117"
Nov 03 18:01:47.472552 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.472525    4493 scope.go:122] "RemoveContainer" containerID="b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117"
Nov 03 18:01:47.473440 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:01:47.473408    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117\": container with ID starting with b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117 not found: ID does not exist" containerID="b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117"
Nov 03 18:01:47.473573 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.473450    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117"} err="failed to get container status \"b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117\": rpc error: code = NotFound desc = could not find container \"b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117\": container with ID starting with b7eed4c394ad8a03c2abf33a185d16da702bc86ff777365cf77f0c8cdfa14117 not found: ID does not exist"
Nov 03 18:01:47.628300 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.628201    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"]
Nov 03 18:01:47.636417 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.636385    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12a446f9-84844b8cb7-mwwdp"]
Nov 03 18:01:47.754790 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.754755    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"]
Nov 03 18:01:47.761611 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:47.761575    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12a446f9-6fd667687c-p7tph"]
Nov 03 18:01:48.713756 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:48.713726    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70b1805-be63-44c3-b924-ce55116bfd49" path="/var/lib/kubelet/pods/d70b1805-be63-44c3-b924-ce55116bfd49/volumes"
Nov 03 18:01:48.714229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:48.714214    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" path="/var/lib/kubelet/pods/f3c4b715-d6e7-4167-b98c-1c3c0992f9b2/volumes"
Nov 03 18:01:50.062853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.062820    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"]
Nov 03 18:01:50.113707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.113668    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc65f59e-lkwhn"]
Nov 03 18:01:50.703573 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.703544    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"]
Nov 03 18:01:50.704168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704154    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" containerName="payload-pre-processing"
Nov 03 18:01:50.704230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704169    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" containerName="payload-pre-processing"
Nov 03 18:01:50.704230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704185    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d70b1805-be63-44c3-b924-ce55116bfd49" containerName="payload-processing"
Nov 03 18:01:50.704230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704190    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70b1805-be63-44c3-b924-ce55116bfd49" containerName="payload-processing"
Nov 03 18:01:50.704230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704209    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" containerName="revoke-keys"
Nov 03 18:01:50.704230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704214    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" containerName="revoke-keys"
Nov 03 18:01:50.704394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704300    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c4b715-d6e7-4167-b98c-1c3c0992f9b2" containerName="payload-pre-processing"
Nov 03 18:01:50.704394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704311    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70b1805-be63-44c3-b924-ce55116bfd49" containerName="payload-processing"
Nov 03 18:01:50.704394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.704319    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" containerName="revoke-keys"
Nov 03 18:01:50.710541 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.710512    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.719825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.719795    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40300b62-552c-47db-81fe-5e99065ae2d1" path="/var/lib/kubelet/pods/40300b62-552c-47db-81fe-5e99065ae2d1/volumes"
Nov 03 18:01:50.767755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.767724    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"]
Nov 03 18:01:50.811558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.811511    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:01:50.811558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.811515    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:01:50.811558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.811530    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:01:50.818412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.818383    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:01:50.855336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.855305    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq89x\" (UniqueName: \"kubernetes.io/projected/cfbbdfce-16d4-485a-9429-148fc050a9e6-kube-api-access-fq89x\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.855493 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.855460    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.855749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.855731    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.957100 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.957011    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.957277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.957202    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.957277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.957262    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq89x\" (UniqueName: \"kubernetes.io/projected/cfbbdfce-16d4-485a-9429-148fc050a9e6-kube-api-access-fq89x\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.957823 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.957799    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:50.959844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:50.959809    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:51.021146 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:51.021105    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq89x\" (UniqueName: \"kubernetes.io/projected/cfbbdfce-16d4-485a-9429-148fc050a9e6-kube-api-access-fq89x\") pod \"maas-api-revoke-sub-4a0042ef-zlf6g\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:51.026061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:51.026032    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:51.411071 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:01:51.411043    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbbdfce_16d4_485a_9429_148fc050a9e6.slice/crio-f1e352975cf84197a513b9635ce6813c476b38de8d9d61139eb0cc5a03341409 WatchSource:0}: Error finding container f1e352975cf84197a513b9635ce6813c476b38de8d9d61139eb0cc5a03341409: Status 404 returned error can't find the container with id f1e352975cf84197a513b9635ce6813c476b38de8d9d61139eb0cc5a03341409
Nov 03 18:01:51.411708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:51.411686    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"]
Nov 03 18:01:51.483235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:51.483192    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g" event={"ID":"cfbbdfce-16d4-485a-9429-148fc050a9e6","Type":"ContainerStarted","Data":"f1e352975cf84197a513b9635ce6813c476b38de8d9d61139eb0cc5a03341409"}
Nov 03 18:01:52.493101 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:52.493065    4493 generic.go:356] "Generic (PLEG): container finished" podID="cfbbdfce-16d4-485a-9429-148fc050a9e6" containerID="35c8e9b146b260abb029ede2f5f3f90f13c24c467b4d43607aeb62c1a02b0e62" exitCode=0
Nov 03 18:01:52.493612 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:52.493112    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g" event={"ID":"cfbbdfce-16d4-485a-9429-148fc050a9e6","Type":"ContainerDied","Data":"35c8e9b146b260abb029ede2f5f3f90f13c24c467b4d43607aeb62c1a02b0e62"}
Nov 03 18:01:53.655371 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.655340    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:53.800660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.800569    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-openshift-ca\") pod \"cfbbdfce-16d4-485a-9429-148fc050a9e6\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") "
Nov 03 18:01:53.800830 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.800659    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-xks-ca\") pod \"cfbbdfce-16d4-485a-9429-148fc050a9e6\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") "
Nov 03 18:01:53.800830 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.800728    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfbbdfce-16d4-485a-9429-148fc050a9e6-kube-api-access-fq89x\" (UniqueName: \"kubernetes.io/projected/cfbbdfce-16d4-485a-9429-148fc050a9e6-kube-api-access-fq89x\") pod \"cfbbdfce-16d4-485a-9429-148fc050a9e6\" (UID: \"cfbbdfce-16d4-485a-9429-148fc050a9e6\") "
Nov 03 18:01:53.801016 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.800981    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-openshift-ca" pod "cfbbdfce-16d4-485a-9429-148fc050a9e6" (UID: "cfbbdfce-16d4-485a-9429-148fc050a9e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:01:53.801445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.801425    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:53.803468 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.803427    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-xks-ca" pod "cfbbdfce-16d4-485a-9429-148fc050a9e6" (UID: "cfbbdfce-16d4-485a-9429-148fc050a9e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:01:53.803809 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.803787    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbbdfce-16d4-485a-9429-148fc050a9e6-kube-api-access-fq89x" pod "cfbbdfce-16d4-485a-9429-148fc050a9e6" (UID: "cfbbdfce-16d4-485a-9429-148fc050a9e6"). InnerVolumeSpecName "kube-api-access-fq89x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:01:53.902976 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.902936    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbbdfce-16d4-485a-9429-148fc050a9e6-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:53.902976 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:53.902965    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq89x\" (UniqueName: \"kubernetes.io/projected/cfbbdfce-16d4-485a-9429-148fc050a9e6-kube-api-access-fq89x\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:01:54.511326 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:54.511298    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"
Nov 03 18:01:54.511512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:54.511327    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g" event={"ID":"cfbbdfce-16d4-485a-9429-148fc050a9e6","Type":"ContainerDied","Data":"f1e352975cf84197a513b9635ce6813c476b38de8d9d61139eb0cc5a03341409"}
Nov 03 18:01:54.511512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:01:54.511386    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e352975cf84197a513b9635ce6813c476b38de8d9d61139eb0cc5a03341409"
Nov 03 18:02:02.061088 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:02.061056    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"]
Nov 03 18:02:02.114193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:02.114162    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b70fc0-px8px"]
Nov 03 18:02:02.714377 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:02.714345    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af2d765-7252-4f60-86de-939e5650e087" path="/var/lib/kubelet/pods/6af2d765-7252-4f60-86de-939e5650e087/volumes"
Nov 03 18:02:06.070813 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:06.070758    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"]
Nov 03 18:02:06.125777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:06.125494    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"]
Nov 03 18:02:06.134786 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:06.134749    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-766a9c5e-k47bs"]
Nov 03 18:02:06.134998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:06.134969    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fca5c3d-g2nff"]
Nov 03 18:02:06.714969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:06.714933    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45430aec-4f43-4d58-b907-80f0312db35e" path="/var/lib/kubelet/pods/45430aec-4f43-4d58-b907-80f0312db35e/volumes"
Nov 03 18:02:06.715508 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:06.715492    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad6d457-21d9-45ad-b594-b5cfbcb496d2" path="/var/lib/kubelet/pods/5ad6d457-21d9-45ad-b594-b5cfbcb496d2/volumes"
Nov 03 18:02:06.994273 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:02:06.994186    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:02:07.983890 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:07.983853    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"]
Nov 03 18:02:08.034456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:08.034423    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-12a446f9-475nw"]
Nov 03 18:02:08.715134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:08.715101    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3c893e-9a68-4d4d-bdfb-dbd4308a4781" path="/var/lib/kubelet/pods/0d3c893e-9a68-4d4d-bdfb-dbd4308a4781/volumes"
Nov 03 18:02:09.076586 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:09.076510    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"]
Nov 03 18:02:09.128840 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:09.128802    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5697d93-xttgb"]
Nov 03 18:02:10.063570 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:10.063533    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"]
Nov 03 18:02:10.116525 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:10.116481    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa9226da-rttbm"]
Nov 03 18:02:10.718523 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:10.718491    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34caf5ab-53b1-4b57-928f-8d056efea5ea" path="/var/lib/kubelet/pods/34caf5ab-53b1-4b57-928f-8d056efea5ea/volumes"
Nov 03 18:02:10.719022 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:10.719009    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b425fc-14ba-440e-a596-310a9261dec8" path="/var/lib/kubelet/pods/50b425fc-14ba-440e-a596-310a9261dec8/volumes"
Nov 03 18:02:16.062071 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:16.062026    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"]
Nov 03 18:02:16.114013 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:16.113976    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b54c9df-wb5k4"]
Nov 03 18:02:16.713916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:16.713862    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511d605d-a149-4ab6-a6a2-07c267a474b3" path="/var/lib/kubelet/pods/511d605d-a149-4ab6-a6a2-07c267a474b3/volumes"
Nov 03 18:02:28.066004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:28.065966    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"]
Nov 03 18:02:28.082538 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:28.082498    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7218fd6b-44ldt"]
Nov 03 18:02:28.713781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:28.713748    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c475a0-21f7-4c6b-a1dc-43ba64dd7270" path="/var/lib/kubelet/pods/c0c475a0-21f7-4c6b-a1dc-43ba64dd7270/volumes"
Nov 03 18:02:33.070090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:33.070054    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"]
Nov 03 18:02:33.121775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:33.121746    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499cc594-5559n"]
Nov 03 18:02:34.714251 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:34.714220    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69c8f25-5592-4c7a-9156-521993aa5d54" path="/var/lib/kubelet/pods/a69c8f25-5592-4c7a-9156-521993aa5d54/volumes"
Nov 03 18:02:36.992940 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:02:36.992903    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:02:40.062631 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:40.062596    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"]
Nov 03 18:02:40.119998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:40.119962    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"]
Nov 03 18:02:40.127939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:40.127895    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b69506-wbrq8"]
Nov 03 18:02:40.145537 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:40.145499    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d02c14c3-fm7vd"]
Nov 03 18:02:40.713590 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:40.713557    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03427e23-9290-4ff8-9f8e-188849cc105b" path="/var/lib/kubelet/pods/03427e23-9290-4ff8-9f8e-188849cc105b/volumes"
Nov 03 18:02:40.714034 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:40.714018    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1912ba5f-fa32-4379-b2fd-654ba9e2c004" path="/var/lib/kubelet/pods/1912ba5f-fa32-4379-b2fd-654ba9e2c004/volumes"
Nov 03 18:02:41.378774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:41.378745    4493 scope.go:122] "RemoveContainer" containerID="5b57d50d1efa3062374e57d2066736eda28be43aac0be9d69080545d5d4d5e80"
Nov 03 18:02:41.388999 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:41.388975    4493 scope.go:122] "RemoveContainer" containerID="d8b1c73c8e180b1f5502d187187cf1418c445b69beac8f95f4169da224d32c7f"
Nov 03 18:02:41.399919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:41.399895    4493 scope.go:122] "RemoveContainer" containerID="f77cd1a9be8317ec0e9ee3f82fb2e50403bb06bc8fe333970804fafdb4841b4e"
Nov 03 18:02:41.414021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:41.413987    4493 scope.go:122] "RemoveContainer" containerID="24c154963a9996f84fdcb23e3d7c0c1f46bc75b955541c20961fdcdb2c1eda57"
Nov 03 18:02:41.425045 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:41.425023    4493 scope.go:122] "RemoveContainer" containerID="eda052f40fd1c6c4292d8875eb90be8802d8da31d7ccbdf88568acd4a9a11613"
Nov 03 18:02:42.062107 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:42.062078    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"]
Nov 03 18:02:42.112490 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:42.112458    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b98b5253-lwnjg"]
Nov 03 18:02:42.713429 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:42.713400    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e108ba7-433c-45b5-a659-7aaddd0b3993" path="/var/lib/kubelet/pods/2e108ba7-433c-45b5-a659-7aaddd0b3993/volumes"
Nov 03 18:02:43.063604 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:43.063527    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"]
Nov 03 18:02:43.123415 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:43.123372    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4540545a-znp4n"]
Nov 03 18:02:44.714578 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:44.714546    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37110d7b-d6ce-43c8-9584-2de563362f4c" path="/var/lib/kubelet/pods/37110d7b-d6ce-43c8-9584-2de563362f4c/volumes"
Nov 03 18:02:56.780078 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.779999    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"]
Nov 03 18:02:56.780637 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.780623    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cfbbdfce-16d4-485a-9429-148fc050a9e6" containerName="revoke-keys"
Nov 03 18:02:56.780682 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.780640    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfbbdfce-16d4-485a-9429-148fc050a9e6" containerName="revoke-keys"
Nov 03 18:02:56.780759 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.780750    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfbbdfce-16d4-485a-9429-148fc050a9e6" containerName="revoke-keys"
Nov 03 18:02:56.786862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.786836    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.842726 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.842698    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-394741e0-openshift-default-dockercfg-hxf6x\""
Nov 03 18:02:56.873143 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873111    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.873143 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873145    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5swr\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-kube-api-access-h5swr\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.873337 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873179    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.873444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873425    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.873636 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873608    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a51d890-33b9-40a4-81c3-d2f54f778891-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.873680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873663    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.873734 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873718    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.873922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.873867    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.874054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.874021    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.895794 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.895767    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"]
Nov 03 18:02:56.975331 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975294    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975521 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975400    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975521 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975464    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975521 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975505    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5swr\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-kube-api-access-h5swr\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975695 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975532    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975695 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975673    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975797 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975679    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975860 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975833    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975836    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a51d890-33b9-40a4-81c3-d2f54f778891-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.975970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975940    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.976073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975991    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.976073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.975990    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.976073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.976031    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.976497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.976474    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a51d890-33b9-40a4-81c3-d2f54f778891-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.978066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.978038    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:56.978767 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:56.978746    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:57.040272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:57.040203    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:57.040387 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:57.040292    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5swr\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-kube-api-access-h5swr\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-c54nj\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:57.101222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:57.101195    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:57.545415 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:02:57.545385    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a51d890_33b9_40a4_81c3_d2f54f778891.slice/crio-1ec39e8e79a46213ddf6f690ee24a1e6cce13f54471f95546c7fc9121abbeb56 WatchSource:0}: Error finding container 1ec39e8e79a46213ddf6f690ee24a1e6cce13f54471f95546c7fc9121abbeb56: Status 404 returned error can't find the container with id 1ec39e8e79a46213ddf6f690ee24a1e6cce13f54471f95546c7fc9121abbeb56
Nov 03 18:02:57.546039 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:57.546015    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"]
Nov 03 18:02:57.547927 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:57.547855    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:57.548009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:57.547964    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:57.548009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:57.547996    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:02:58.073330 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:58.073292    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" event={"ID":"3a51d890-33b9-40a4-81c3-d2f54f778891","Type":"ContainerStarted","Data":"4699fe48133688a6fbc9042ec5bf7f7fc5475f7c6a0ebc8a28b0c5e7f390bf6b"}
Nov 03 18:02:58.073750 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:58.073360    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" event={"ID":"3a51d890-33b9-40a4-81c3-d2f54f778891","Type":"ContainerStarted","Data":"1ec39e8e79a46213ddf6f690ee24a1e6cce13f54471f95546c7fc9121abbeb56"}
Nov 03 18:02:58.102305 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:58.102280    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:02:58.103946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:58.103918    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:02:58.104078 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:58.103976    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:02:58.201654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:58.201551    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podStartSLOduration=2.2015362 podStartE2EDuration="2.2015362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:58.199320394 +0000 UTC m=+1757.658491406" watchObservedRunningTime="2026-11-03 18:02:58.2015362 +0000 UTC m=+1757.660707247"
Nov 03 18:02:59.062321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:59.062289    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"]
Nov 03 18:02:59.102324 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:59.102295    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:02:59.102688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:59.102352    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:02:59.135990 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:02:59.135947    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671c255a-jqxr4"]
Nov 03 18:03:00.102225 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:00.102190    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:03:00.102400 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:00.102249    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:03:00.713636 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:00.713605    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3749470-e5ad-49df-bd13-ed8215373bf7" path="/var/lib/kubelet/pods/a3749470-e5ad-49df-bd13-ed8215373bf7/volumes"
Nov 03 18:03:01.080367 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:01.080289    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"]
Nov 03 18:03:01.102431 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:01.102396    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:03:01.102796 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:01.102445    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:03:01.146403 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:01.146371    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0750fdee-9zq4p"]
Nov 03 18:03:02.102394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:02.102356    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:03:02.102599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:02.102403    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:03:02.714063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:02.714025    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a" path="/var/lib/kubelet/pods/1e676c3d-b0e9-4594-9c9d-f2edc5b27c1a/volumes"
Nov 03 18:03:03.102231 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:03.102147    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 18:03:03.102231 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:03.102208    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 18:03:04.107210 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:04.107170    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:03:04.107909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:04.107851    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:03:04.108463 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:04.108441    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:03:06.061215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:06.061169    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"]
Nov 03 18:03:06.113543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:06.113511    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d520831-qczh7"]
Nov 03 18:03:06.719848 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:06.719807    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ed1d0d-85a5-41f3-ae17-a6ce22efc00a" path="/var/lib/kubelet/pods/14ed1d0d-85a5-41f3-ae17-a6ce22efc00a/volumes"
Nov 03 18:03:06.994150 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:03:06.994018    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:03:07.009791 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.001044    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"]
Nov 03 18:03:07.009791 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.006668    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.058973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.058942    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.059149 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.059120    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjld\" (UniqueName: \"kubernetes.io/projected/7f25fc28-8eef-4554-98a8-66ef06c021b5-kube-api-access-rwjld\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.059306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.059293    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.066223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.066197    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"]
Nov 03 18:03:07.118942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.112075    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:03:07.118942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.112161    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:03:07.118942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.112207    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:03:07.118942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.112363    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:03:07.119971 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.119944    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"]
Nov 03 18:03:07.124272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.124248    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4edab22e-gjj4n"]
Nov 03 18:03:07.160920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.160891    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjld\" (UniqueName: \"kubernetes.io/projected/7f25fc28-8eef-4554-98a8-66ef06c021b5-kube-api-access-rwjld\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.161079 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.161066    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.161158 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.161114    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.161813 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.161796    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.163645 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.163623    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.225067 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.225036    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjld\" (UniqueName: \"kubernetes.io/projected/7f25fc28-8eef-4554-98a8-66ef06c021b5-kube-api-access-rwjld\") pod \"maas-api-revoke-keys-e2e-ait-3e138c04-kn588\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.321618 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.321533    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:07.703961 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:03:07.703917    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f25fc28_8eef_4554_98a8_66ef06c021b5.slice/crio-0a5d4be20d9c244d0b71c2345c7db920186fe6a934c373647df8320ed28d5c5e WatchSource:0}: Error finding container 0a5d4be20d9c244d0b71c2345c7db920186fe6a934c373647df8320ed28d5c5e: Status 404 returned error can't find the container with id 0a5d4be20d9c244d0b71c2345c7db920186fe6a934c373647df8320ed28d5c5e
Nov 03 18:03:07.704081 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:07.703989    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"]
Nov 03 18:03:08.279753 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:08.274522    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588" event={"ID":"7f25fc28-8eef-4554-98a8-66ef06c021b5","Type":"ContainerStarted","Data":"f58c57755722cfa8b701278ccc462edc47b046594fbc9bcbce67bc93a7f31bcc"}
Nov 03 18:03:08.279753 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:08.274589    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588" event={"ID":"7f25fc28-8eef-4554-98a8-66ef06c021b5","Type":"ContainerStarted","Data":"0a5d4be20d9c244d0b71c2345c7db920186fe6a934c373647df8320ed28d5c5e"}
Nov 03 18:03:08.714647 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:08.714611    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890b3ab1-3e60-4bad-8ec1-e19ad007d7c0" path="/var/lib/kubelet/pods/890b3ab1-3e60-4bad-8ec1-e19ad007d7c0/volumes"
Nov 03 18:03:09.059329 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:09.059276    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588" podStartSLOduration=3.059260911 podStartE2EDuration="3.059260911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:08.396085896 +0000 UTC m=+1767.855256883" watchObservedRunningTime="2026-11-03 18:03:09.059260911 +0000 UTC m=+1768.518431897"
Nov 03 18:03:09.060705 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:09.060680    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"]
Nov 03 18:03:09.113411 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:09.113370    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c274391f-qt67h"]
Nov 03 18:03:09.283604 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:09.283568    4493 generic.go:356] "Generic (PLEG): container finished" podID="7f25fc28-8eef-4554-98a8-66ef06c021b5" containerID="f58c57755722cfa8b701278ccc462edc47b046594fbc9bcbce67bc93a7f31bcc" exitCode=0
Nov 03 18:03:09.284043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:09.283658    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588" event={"ID":"7f25fc28-8eef-4554-98a8-66ef06c021b5","Type":"ContainerDied","Data":"f58c57755722cfa8b701278ccc462edc47b046594fbc9bcbce67bc93a7f31bcc"}
Nov 03 18:03:10.470422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.470393    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:10.628256 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.628167    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-xks-ca\") pod \"7f25fc28-8eef-4554-98a8-66ef06c021b5\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") "
Nov 03 18:03:10.628439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.628306    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-openshift-ca\") pod \"7f25fc28-8eef-4554-98a8-66ef06c021b5\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") "
Nov 03 18:03:10.628439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.628425    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f25fc28-8eef-4554-98a8-66ef06c021b5-kube-api-access-rwjld\" (UniqueName: \"kubernetes.io/projected/7f25fc28-8eef-4554-98a8-66ef06c021b5-kube-api-access-rwjld\") pod \"7f25fc28-8eef-4554-98a8-66ef06c021b5\" (UID: \"7f25fc28-8eef-4554-98a8-66ef06c021b5\") "
Nov 03 18:03:10.628765 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.628736    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-openshift-ca" pod "7f25fc28-8eef-4554-98a8-66ef06c021b5" (UID: "7f25fc28-8eef-4554-98a8-66ef06c021b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:03:10.629368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.629339    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:10.630693 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.630662    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-xks-ca" pod "7f25fc28-8eef-4554-98a8-66ef06c021b5" (UID: "7f25fc28-8eef-4554-98a8-66ef06c021b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:03:10.631007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.630978    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f25fc28-8eef-4554-98a8-66ef06c021b5-kube-api-access-rwjld" pod "7f25fc28-8eef-4554-98a8-66ef06c021b5" (UID: "7f25fc28-8eef-4554-98a8-66ef06c021b5"). InnerVolumeSpecName "kube-api-access-rwjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:10.714732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.714700    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f720ee-ecfe-4e21-a2c3-ffe0519d901a" path="/var/lib/kubelet/pods/10f720ee-ecfe-4e21-a2c3-ffe0519d901a/volumes"
Nov 03 18:03:10.731060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.731033    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwjld\" (UniqueName: \"kubernetes.io/projected/7f25fc28-8eef-4554-98a8-66ef06c021b5-kube-api-access-rwjld\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:10.731174 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:10.731060    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f25fc28-8eef-4554-98a8-66ef06c021b5-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:11.299957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.299927    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"
Nov 03 18:03:11.299957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.299943    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588" event={"ID":"7f25fc28-8eef-4554-98a8-66ef06c021b5","Type":"ContainerDied","Data":"0a5d4be20d9c244d0b71c2345c7db920186fe6a934c373647df8320ed28d5c5e"}
Nov 03 18:03:11.300290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.299998    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a5d4be20d9c244d0b71c2345c7db920186fe6a934c373647df8320ed28d5c5e"
Nov 03 18:03:11.856808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.856768    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"]
Nov 03 18:03:11.857687 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.857663    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7f25fc28-8eef-4554-98a8-66ef06c021b5" containerName="revoke-keys"
Nov 03 18:03:11.857687 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.857687    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f25fc28-8eef-4554-98a8-66ef06c021b5" containerName="revoke-keys"
Nov 03 18:03:11.857887 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.857859    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f25fc28-8eef-4554-98a8-66ef06c021b5" containerName="revoke-keys"
Nov 03 18:03:11.861925 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.861865    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.906473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.906443    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"]
Nov 03 18:03:11.910982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.910961    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:11.951502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951467    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6995g\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-kube-api-access-6995g\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.951666 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951550    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.951666 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951587    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.951758 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951663    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.951758 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951689    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.951758 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951740    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8341c4-0eb6-489f-b16e-d190ba76201f-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.951938 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951779    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.952020 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.951995    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.952263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.952241    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:11.979972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.979934    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"]
Nov 03 18:03:11.986503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:11.986466    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"]
Nov 03 18:03:12.054189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054159    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054194    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add8c803-d76c-42e5-99a7-e4c41824e10e-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054229    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054349    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054468 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054413    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.054468 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054438    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054487    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.054532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054527    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6995g\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-kube-api-access-6995g\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.054598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054542    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkp95\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-kube-api-access-gkp95\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054592    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054666 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054649    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.054701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054682    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.054735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054716    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.054838 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054815    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.054944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054860    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.055013 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054942    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8341c4-0eb6-489f-b16e-d190ba76201f-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.055013 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.054972    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.055125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.055008    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.055125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.055012    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.055125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.055070    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.055125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.055087    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.055335 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.055316    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.055605 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.055590    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8341c4-0eb6-489f-b16e-d190ba76201f-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.057128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.057105    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.057240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.057217    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.123953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.123712    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"]
Nov 03 18:03:12.124223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.124198    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.127549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.127519    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6995g\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-kube-api-access-6995g\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-56rr4\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.138190 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.138168    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.156413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156387    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156440    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156574 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156524    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156651 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156637    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkp95\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-kube-api-access-gkp95\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156715 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156681    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156769 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156728    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156821 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156794    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156910    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.156993 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.156912    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.157041 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.157025    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.157088 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.157060    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add8c803-d76c-42e5-99a7-e4c41824e10e-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.157146 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.157126    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.157377 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.157351    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.157699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.157678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add8c803-d76c-42e5-99a7-e4c41824e10e-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.159033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.159013    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.159548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.159528    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.176289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.176270    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:12.189947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.189920    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"]
Nov 03 18:03:12.223455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.223421    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.224144 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.224123    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkp95\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-kube-api-access-gkp95\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.227028 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.227005    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:12.259197 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259111    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259279 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259242    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259300    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259403 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259355    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259403 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259391    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259442    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2hg\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-kube-api-access-6w2hg\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259554 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259513    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259627    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.259996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.259671    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.361945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.361908    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.361964    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.362010    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2hg\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-kube-api-access-6w2hg\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.362072    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362335 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.362176    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362335 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.362215    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.362377    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.362519    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.362599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.362573    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.363455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.363422    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.363726 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.363701    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.365007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.364982    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.365238 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.365215    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.365902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.365821    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.366528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.366504    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.367300 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.367271    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.428043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.427976    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.428043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.428027    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2hg\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-kube-api-access-6w2hg\") pod \"maas-default-gateway-openshift-default-5df48fcc99-hjbns\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.461135 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.461110    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:12.633618 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:03:12.633591    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8341c4_0eb6_489f_b16e_d190ba76201f.slice/crio-12083be50631365568478fc302eb48ce4575bce03217ed514f34c0814bd5c310 WatchSource:0}: Error finding container 12083be50631365568478fc302eb48ce4575bce03217ed514f34c0814bd5c310: Status 404 returned error can't find the container with id 12083be50631365568478fc302eb48ce4575bce03217ed514f34c0814bd5c310
Nov 03 18:03:12.634721 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.634692    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"]
Nov 03 18:03:12.636370 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.636339    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.636465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.636403    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.636465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.636434    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.685930 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:03:12.685902    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd8c803_d76c_42e5_99a7_e4c41824e10e.slice/crio-8e6e31cb15e754c83288a9d884808651bc6014c5e1470a67b24790c5c12f85ad WatchSource:0}: Error finding container 8e6e31cb15e754c83288a9d884808651bc6014c5e1470a67b24790c5c12f85ad: Status 404 returned error can't find the container with id 8e6e31cb15e754c83288a9d884808651bc6014c5e1470a67b24790c5c12f85ad
Nov 03 18:03:12.688191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.688165    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"]
Nov 03 18:03:12.688769 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.688732    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.688856 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.688819    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.688951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.688863    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.852170 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.852146    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"]
Nov 03 18:03:12.859752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.859716    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.860194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.859800    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:12.860194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:12.859850    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:03:13.321392 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.321352    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" event={"ID":"5b8341c4-0eb6-489f-b16e-d190ba76201f","Type":"ContainerStarted","Data":"71496b777fc1bad0521b6c3c2d43445b0120a875ca4c743d7848420fc4d5ac0d"}
Nov 03 18:03:13.321574 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.321429    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" event={"ID":"5b8341c4-0eb6-489f-b16e-d190ba76201f","Type":"ContainerStarted","Data":"12083be50631365568478fc302eb48ce4575bce03217ed514f34c0814bd5c310"}
Nov 03 18:03:13.323327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.323296    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" event={"ID":"9bcac327-d2fe-4d33-bc16-ebd373d687f3","Type":"ContainerStarted","Data":"c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609"}
Nov 03 18:03:13.323480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.323347    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" event={"ID":"9bcac327-d2fe-4d33-bc16-ebd373d687f3","Type":"ContainerStarted","Data":"6ee4cc83bd55cd3453025f3f456c6e87230432e8b7a54d2542fde4e56fa501b5"}
Nov 03 18:03:13.325144 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.325119    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" event={"ID":"add8c803-d76c-42e5-99a7-e4c41824e10e","Type":"ContainerStarted","Data":"a75df992e1f7cca5a57617ea165768fc3cc2bb05488df02d3d91dc5e56e4fe9f"}
Nov 03 18:03:13.325263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.325164    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" event={"ID":"add8c803-d76c-42e5-99a7-e4c41824e10e","Type":"ContainerStarted","Data":"8e6e31cb15e754c83288a9d884808651bc6014c5e1470a67b24790c5c12f85ad"}
Nov 03 18:03:13.451193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.451141    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podStartSLOduration=2.451124716 podStartE2EDuration="2.451124716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:13.447186139 +0000 UTC m=+1772.906357151" watchObservedRunningTime="2026-11-03 18:03:13.451124716 +0000 UTC m=+1772.910295702"
Nov 03 18:03:13.462272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.462225    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:13.463911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.463860    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:13.464074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.463939    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:13.580314 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.580208    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podStartSLOduration=1.5801931790000001 podStartE2EDuration="1.580193179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:13.576716172 +0000 UTC m=+1773.035887158" watchObservedRunningTime="2026-11-03 18:03:13.580193179 +0000 UTC m=+1773.039364164"
Nov 03 18:03:13.776281 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:13.776207    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podStartSLOduration=2.7761652420000003 podStartE2EDuration="2.776165242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:13.771603089 +0000 UTC m=+1773.230774076" watchObservedRunningTime="2026-11-03 18:03:13.776165242 +0000 UTC m=+1773.235336236"
Nov 03 18:03:14.188902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.188075    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:14.199907 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.192977    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 03 18:03:14.199907 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.193035    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 03 18:03:14.232899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.227468    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:14.232899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.231832    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:03:14.232899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.231910    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:03:14.462959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.462312    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:14.462959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:14.462373    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:15.178520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.178471    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 03 18:03:15.178889 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.178838    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 03 18:03:15.244029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.243988    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:03:15.244511 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.244069    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:03:15.462957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.462328    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:15.462957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.462390    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:15.631775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.631721    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"]
Nov 03 18:03:15.678297 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.678264    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:15.701406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.701363    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"]
Nov 03 18:03:15.750962 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.745282    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:03:15.750962 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.746033    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:03:15.750962 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.746134    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:03:15.750962 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.746184    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:03:15.904910 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.904854    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:15.905217 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.905198    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:15.905580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:15.905562    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7x2f\" (UniqueName: \"kubernetes.io/projected/a31d690d-0e4e-4f42-8530-d071abb293ff-kube-api-access-z7x2f\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.007083 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.006996    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.007273 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.007250    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7x2f\" (UniqueName: \"kubernetes.io/projected/a31d690d-0e4e-4f42-8530-d071abb293ff-kube-api-access-z7x2f\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.007384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.007369    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.008142 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.008120    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.010000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.009976    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.083903 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.083627    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7x2f\" (UniqueName: \"kubernetes.io/projected/a31d690d-0e4e-4f42-8530-d071abb293ff-kube-api-access-z7x2f\") pod \"maas-api-revoke-keys-e2e-ait-394741e0-9gqv5\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.098898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.096840    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:16.209108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.209064    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 03 18:03:16.209317 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.209149    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 03 18:03:16.231291 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.231251    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:03:16.231572 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.231539    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:03:16.499773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.499302    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:16.499773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.499398    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:16.693983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:16.693349    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"]
Nov 03 18:03:16.701812 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:03:16.701778    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31d690d_0e4e_4f42_8530_d071abb293ff.slice/crio-dd7d2bfa5fe301bb30ee50bd24a31dabc56539c0d77788d1a71278c6c6da73e3 WatchSource:0}: Error finding container dd7d2bfa5fe301bb30ee50bd24a31dabc56539c0d77788d1a71278c6c6da73e3: Status 404 returned error can't find the container with id dd7d2bfa5fe301bb30ee50bd24a31dabc56539c0d77788d1a71278c6c6da73e3
Nov 03 18:03:17.190897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:17.188949    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 03 18:03:17.190897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:17.189006    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 03 18:03:17.230902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:17.230241    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:03:17.230902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:17.230302    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:03:17.594094 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:17.594057    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:17.595344 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:17.595312    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:18.061898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.058509    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5" event={"ID":"a31d690d-0e4e-4f42-8530-d071abb293ff","Type":"ContainerStarted","Data":"dd7d2bfa5fe301bb30ee50bd24a31dabc56539c0d77788d1a71278c6c6da73e3"}
Nov 03 18:03:18.161531 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.161487    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:18.183840 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.180537    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 18:03:18.196617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.196581    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 03 18:03:18.218831 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.218787    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 03 18:03:18.219123 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.196840    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:18.242007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.241963    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:03:18.242279 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.242246    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:03:18.426956 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.401338    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:18.455987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.455955    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"]
Nov 03 18:03:18.480030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.479994    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:18.480307 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.480283    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:18.480453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.480439    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ec42eb2-45dlk"]
Nov 03 18:03:18.510419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.507180    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5" podStartSLOduration=3.507156401 podStartE2EDuration="3.507156401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:18.218711274 +0000 UTC m=+1777.677882260" watchObservedRunningTime="2026-11-03 18:03:18.507156401 +0000 UTC m=+1777.966327389"
Nov 03 18:03:18.894931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.847963    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 18:03:18.894931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.848305    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:18.937576 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:18.922349    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:19.067361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.067306    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:19.098402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.098309    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 18:03:19.116351 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.115517    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab632e2-f694-46a6-ac8c-df8c1f34b7dd" path="/var/lib/kubelet/pods/eab632e2-f694-46a6-ac8c-df8c1f34b7dd/volumes"
Nov 03 18:03:19.190956 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.188580    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Nov 03 18:03:19.190956 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.188652    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Nov 03 18:03:19.193903 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.193557    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:19.206274 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.196353    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 18:03:19.322910 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.300409    4493 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 18:03:19.323193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.323155    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 18:03:19.354058 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.354012    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:03:19.509900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.470316    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:19.509900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.470377    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:19.656899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.652593    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 18:03:19.698338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.698298    4493 generic.go:356] "Generic (PLEG): container finished" podID="a31d690d-0e4e-4f42-8530-d071abb293ff" containerID="c519d74a3bb3bd4fdb8e85c3b2165cf68aed9d92f4d3cbe519bab0ed9f23507e" exitCode=0
Nov 03 18:03:19.698806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:19.698586    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5" event={"ID":"a31d690d-0e4e-4f42-8530-d071abb293ff","Type":"ContainerDied","Data":"c519d74a3bb3bd4fdb8e85c3b2165cf68aed9d92f4d3cbe519bab0ed9f23507e"}
Nov 03 18:03:20.227900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.214194    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:20.227900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.215904    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:20.244912 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.244714    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:03:20.264811 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.260143    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:20.264811 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.261425    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:20.264811 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.262109    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:03:20.461559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.461527    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:20.461721 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.461580    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:20.968306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.968283    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:20.976221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.974303    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31d690d-0e4e-4f42-8530-d071abb293ff-kube-api-access-z7x2f\" (UniqueName: \"kubernetes.io/projected/a31d690d-0e4e-4f42-8530-d071abb293ff-kube-api-access-z7x2f\") pod \"a31d690d-0e4e-4f42-8530-d071abb293ff\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") "
Nov 03 18:03:20.976221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.974447    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-xks-ca\") pod \"a31d690d-0e4e-4f42-8530-d071abb293ff\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") "
Nov 03 18:03:20.976221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.974578    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-openshift-ca\") pod \"a31d690d-0e4e-4f42-8530-d071abb293ff\" (UID: \"a31d690d-0e4e-4f42-8530-d071abb293ff\") "
Nov 03 18:03:20.976221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.975578    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-openshift-ca" pod "a31d690d-0e4e-4f42-8530-d071abb293ff" (UID: "a31d690d-0e4e-4f42-8530-d071abb293ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:03:20.979762 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.979733    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31d690d-0e4e-4f42-8530-d071abb293ff-kube-api-access-z7x2f" pod "a31d690d-0e4e-4f42-8530-d071abb293ff" (UID: "a31d690d-0e4e-4f42-8530-d071abb293ff"). InnerVolumeSpecName "kube-api-access-z7x2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:20.981686 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:20.981661    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-xks-ca" pod "a31d690d-0e4e-4f42-8530-d071abb293ff" (UID: "a31d690d-0e4e-4f42-8530-d071abb293ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:03:21.065497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.065463    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"]
Nov 03 18:03:21.069321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.069297    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e477a476-n8fm8"]
Nov 03 18:03:21.076258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.076231    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7x2f\" (UniqueName: \"kubernetes.io/projected/a31d690d-0e4e-4f42-8530-d071abb293ff-kube-api-access-z7x2f\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:21.076258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.076259    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:21.076563 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.076269    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d690d-0e4e-4f42-8530-d071abb293ff-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:21.462072 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.461995    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 18:03:21.462072 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.462051    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 18:03:21.719017 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.718983    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"
Nov 03 18:03:21.719188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.719004    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5" event={"ID":"a31d690d-0e4e-4f42-8530-d071abb293ff","Type":"ContainerDied","Data":"dd7d2bfa5fe301bb30ee50bd24a31dabc56539c0d77788d1a71278c6c6da73e3"}
Nov 03 18:03:21.719188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:21.719066    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd7d2bfa5fe301bb30ee50bd24a31dabc56539c0d77788d1a71278c6c6da73e3"
Nov 03 18:03:22.719084 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:22.719040    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66e698d-7f14-43b2-8758-a09b1d16dbd3" path="/var/lib/kubelet/pods/e66e698d-7f14-43b2-8758-a09b1d16dbd3/volumes"
Nov 03 18:03:22.943094 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:22.943059    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:22.943563 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:22.943538    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:22.945574 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:22.945547    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:03:26.067396 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:26.067363    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"]
Nov 03 18:03:26.121927 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:26.121860    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f6ea29-btxcn"]
Nov 03 18:03:26.718676 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:26.718642    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0f0c85-2e01-4005-9132-85df6d047cdd" path="/var/lib/kubelet/pods/7b0f0c85-2e01-4005-9132-85df6d047cdd/volumes"
Nov 03 18:03:32.225134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.225091    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"]
Nov 03 18:03:32.225957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.225936    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a31d690d-0e4e-4f42-8530-d071abb293ff" containerName="revoke-keys"
Nov 03 18:03:32.226077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.225957    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31d690d-0e4e-4f42-8530-d071abb293ff" containerName="revoke-keys"
Nov 03 18:03:32.226186 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.226171    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31d690d-0e4e-4f42-8530-d071abb293ff" containerName="revoke-keys"
Nov 03 18:03:32.229591 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.229566    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.284943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.284907    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:03:32.315777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.315298    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.315777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.315405    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.315777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.315605    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc5xm\" (UniqueName: \"kubernetes.io/projected/7eae40ab-5a22-414f-9792-82b31b272ecc-kube-api-access-cc5xm\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.334707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.334671    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:03:32.334896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.334714    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:03:32.334896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.334742    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:03:32.339200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.339170    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"]
Nov 03 18:03:32.418804 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.418767    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc5xm\" (UniqueName: \"kubernetes.io/projected/7eae40ab-5a22-414f-9792-82b31b272ecc-kube-api-access-cc5xm\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.419010 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.418947    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.419769 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.419740    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.419928 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.419909    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.422547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.422521    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.484455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.484386    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc5xm\" (UniqueName: \"kubernetes.io/projected/7eae40ab-5a22-414f-9792-82b31b272ecc-kube-api-access-cc5xm\") pod \"maas-api-revoke-sub-5c430873-5pdbn\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.543242 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.543212    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:32.941018 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:32.940923    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"]
Nov 03 18:03:32.947472 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:03:32.947445    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eae40ab_5a22_414f_9792_82b31b272ecc.slice/crio-37940c6cc00de825ead43d21b77e601e08fa0e46fb2b49d49da69ceb41c82de5 WatchSource:0}: Error finding container 37940c6cc00de825ead43d21b77e601e08fa0e46fb2b49d49da69ceb41c82de5: Status 404 returned error can't find the container with id 37940c6cc00de825ead43d21b77e601e08fa0e46fb2b49d49da69ceb41c82de5
Nov 03 18:03:33.864886 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:33.864843    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn" event={"ID":"7eae40ab-5a22-414f-9792-82b31b272ecc","Type":"ContainerStarted","Data":"92e06e25bea80c98317489e6a8ad8ae3ba0141f08583fb0fa0c4bd0e2040ede6"}
Nov 03 18:03:33.865360 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:33.864926    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn" event={"ID":"7eae40ab-5a22-414f-9792-82b31b272ecc","Type":"ContainerStarted","Data":"37940c6cc00de825ead43d21b77e601e08fa0e46fb2b49d49da69ceb41c82de5"}
Nov 03 18:03:33.986657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:33.986607    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn" podStartSLOduration=1.986591603 podStartE2EDuration="1.986591603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:33.984315676 +0000 UTC m=+1793.443486664" watchObservedRunningTime="2026-11-03 18:03:33.986591603 +0000 UTC m=+1793.445762662"
Nov 03 18:03:34.875543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:34.875506    4493 generic.go:356] "Generic (PLEG): container finished" podID="7eae40ab-5a22-414f-9792-82b31b272ecc" containerID="92e06e25bea80c98317489e6a8ad8ae3ba0141f08583fb0fa0c4bd0e2040ede6" exitCode=0
Nov 03 18:03:34.875961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:34.875587    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn" event={"ID":"7eae40ab-5a22-414f-9792-82b31b272ecc","Type":"ContainerDied","Data":"92e06e25bea80c98317489e6a8ad8ae3ba0141f08583fb0fa0c4bd0e2040ede6"}
Nov 03 18:03:36.073253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.073213    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:36.082622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.082593    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-openshift-ca\") pod \"7eae40ab-5a22-414f-9792-82b31b272ecc\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") "
Nov 03 18:03:36.082754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.082695    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eae40ab-5a22-414f-9792-82b31b272ecc-kube-api-access-cc5xm\" (UniqueName: \"kubernetes.io/projected/7eae40ab-5a22-414f-9792-82b31b272ecc-kube-api-access-cc5xm\") pod \"7eae40ab-5a22-414f-9792-82b31b272ecc\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") "
Nov 03 18:03:36.082845 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.082787    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-xks-ca\") pod \"7eae40ab-5a22-414f-9792-82b31b272ecc\" (UID: \"7eae40ab-5a22-414f-9792-82b31b272ecc\") "
Nov 03 18:03:36.082988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.082961    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-openshift-ca" pod "7eae40ab-5a22-414f-9792-82b31b272ecc" (UID: "7eae40ab-5a22-414f-9792-82b31b272ecc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:03:36.083508 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.083488    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:36.085117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.085089    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eae40ab-5a22-414f-9792-82b31b272ecc-kube-api-access-cc5xm" pod "7eae40ab-5a22-414f-9792-82b31b272ecc" (UID: "7eae40ab-5a22-414f-9792-82b31b272ecc"). InnerVolumeSpecName "kube-api-access-cc5xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:03:36.085228 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.085180    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-xks-ca" pod "7eae40ab-5a22-414f-9792-82b31b272ecc" (UID: "7eae40ab-5a22-414f-9792-82b31b272ecc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:03:36.185662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.185580    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc5xm\" (UniqueName: \"kubernetes.io/projected/7eae40ab-5a22-414f-9792-82b31b272ecc-kube-api-access-cc5xm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:36.185662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.185612    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eae40ab-5a22-414f-9792-82b31b272ecc-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:03:36.893582 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.893550    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"
Nov 03 18:03:36.893756 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.893552    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn" event={"ID":"7eae40ab-5a22-414f-9792-82b31b272ecc","Type":"ContainerDied","Data":"37940c6cc00de825ead43d21b77e601e08fa0e46fb2b49d49da69ceb41c82de5"}
Nov 03 18:03:36.893756 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:36.893674    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37940c6cc00de825ead43d21b77e601e08fa0e46fb2b49d49da69ceb41c82de5"
Nov 03 18:03:36.993977 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:03:36.993892    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:03:40.061499 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:40.061463    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"]
Nov 03 18:03:40.114263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:40.114230    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-238b4a52-xrjxt"]
Nov 03 18:03:40.715106 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:40.715071    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22da52e-9360-4ca9-b465-9d2a502f8281" path="/var/lib/kubelet/pods/c22da52e-9360-4ca9-b465-9d2a502f8281/volumes"
Nov 03 18:03:41.062334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.062241    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"]
Nov 03 18:03:41.118768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.118732    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d01cf4e8-n5klw"]
Nov 03 18:03:41.483998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.483972    4493 scope.go:122] "RemoveContainer" containerID="43609cab1051d412e76db2a53548ddec132ef23f27ef9201a77118e754ea14de"
Nov 03 18:03:41.495706 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.495687    4493 scope.go:122] "RemoveContainer" containerID="5fc0d0ba39e86f879194517f7fa8c37c776b5dfae40e1f9a5d7c1e38dd3ba50d"
Nov 03 18:03:41.507084 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.507064    4493 scope.go:122] "RemoveContainer" containerID="8523777bf209bf4395039bb4d396c6745239b35246d53d13b91dd983fd10c715"
Nov 03 18:03:41.517644 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.517620    4493 scope.go:122] "RemoveContainer" containerID="7a78b5baffec4a3195f0842aacb91b6d6a9ceedb236ef17b4871dbf1067d9212"
Nov 03 18:03:41.527341 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.527314    4493 scope.go:122] "RemoveContainer" containerID="92ab83ec2712b31586a00bee18e7b2c9d26477c227fbcd2c1decc534fdeff505"
Nov 03 18:03:41.536459 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.536436    4493 scope.go:122] "RemoveContainer" containerID="ee34b46c05e34dce936f274873b29b4db52c4455cf2b6b831552106e3fedb614"
Nov 03 18:03:41.545168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.545137    4493 scope.go:122] "RemoveContainer" containerID="fafd3b4e924262db6d126bccbed73a1ae2d4da0e10fee8442d819685ef50c59c"
Nov 03 18:03:41.553944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.553923    4493 scope.go:122] "RemoveContainer" containerID="3da44fe28b5b1e58ef545a0aed763c2384bb26f454bfca5661910e3bbd4532a3"
Nov 03 18:03:41.562658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.562642    4493 scope.go:122] "RemoveContainer" containerID="c3707ca42148ff62aa8a444b01fe5ee966aaff556823f091aac5466fad40beac"
Nov 03 18:03:41.571452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.571437    4493 scope.go:122] "RemoveContainer" containerID="9ae84cc79a9fc91a36a388d8a4821a5e5609a93f0fabd8612a35e8360351522f"
Nov 03 18:03:41.580981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.580964    4493 scope.go:122] "RemoveContainer" containerID="287d65364004310e7d3f32769dd09d5da89c70ea863d41d5eb5ceb41ea9d8042"
Nov 03 18:03:41.589614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.589596    4493 scope.go:122] "RemoveContainer" containerID="171354be469ecf1264df6003b45112acd21eb91b625eeda938dfa5f90e1bf3ab"
Nov 03 18:03:41.601941 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.601920    4493 scope.go:122] "RemoveContainer" containerID="d76362e9c85352cf3b14e6d543acc42069f18295f5dcf92e95cdf963ea395433"
Nov 03 18:03:41.611472 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:41.611451    4493 scope.go:122] "RemoveContainer" containerID="35c701d9992865d3c33d1c6b9247ac3a3f8458b43135f15f57c373de117c5290"
Nov 03 18:03:42.715262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:42.715222    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1646af9-12a5-4a66-8e23-89435dc506dc" path="/var/lib/kubelet/pods/a1646af9-12a5-4a66-8e23-89435dc506dc/volumes"
Nov 03 18:03:52.061724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:52.061686    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"]
Nov 03 18:03:52.114216 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:52.114178    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb093433-587kf"]
Nov 03 18:03:52.118099 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:52.118072    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"]
Nov 03 18:03:52.121923 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:52.121898    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e986bb5b-d7jfq"]
Nov 03 18:03:52.713438 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:52.713410    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45cdf8f-e31c-46fb-b29a-7b82d23ced50" path="/var/lib/kubelet/pods/c45cdf8f-e31c-46fb-b29a-7b82d23ced50/volumes"
Nov 03 18:03:52.713883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:03:52.713856    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa899b9b-aa21-4fd6-8f52-ab44711e2996" path="/var/lib/kubelet/pods/fa899b9b-aa21-4fd6-8f52-ab44711e2996/volumes"
Nov 03 18:04:06.982334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:06.982299    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"]
Nov 03 18:04:06.996531 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:04:06.996500    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:04:07.032609 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:07.032572    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3e138c04-kn588"]
Nov 03 18:04:08.170296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.170255    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"]
Nov 03 18:04:08.218825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.218790    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-394741e0-9gqv5"]
Nov 03 18:04:08.976475 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.975888    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f25fc28-8eef-4554-98a8-66ef06c021b5" path="/var/lib/kubelet/pods/7f25fc28-8eef-4554-98a8-66ef06c021b5/volumes"
Nov 03 18:04:08.977027 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.977001    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31d690d-0e4e-4f42-8530-d071abb293ff" path="/var/lib/kubelet/pods/a31d690d-0e4e-4f42-8530-d071abb293ff/volumes"
Nov 03 18:04:08.977707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.977678    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"]
Nov 03 18:04:08.980123 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.980097    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy" containerID="cri-o://4699fe48133688a6fbc9042ec5bf7f7fc5475f7c6a0ebc8a28b0c5e7f390bf6b" gracePeriod=30
Nov 03 18:04:08.983592 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.983573    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"]
Nov 03 18:04:08.984001 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.983976    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy" containerID="cri-o://a75df992e1f7cca5a57617ea165768fc3cc2bb05488df02d3d91dc5e56e4fe9f" gracePeriod=30
Nov 03 18:04:08.993360 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.993337    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"]
Nov 03 18:04:08.993744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:08.993723    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy" containerID="cri-o://71496b777fc1bad0521b6c3c2d43445b0120a875ca4c743d7848420fc4d5ac0d" gracePeriod=30
Nov 03 18:04:12.327617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:12.327583    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"]
Nov 03 18:04:12.328143 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:12.327846    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy" containerID="cri-o://a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69" gracePeriod=30
Nov 03 18:04:13.281602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:13.281555    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"]
Nov 03 18:04:13.281857 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:13.281828    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy" containerID="cri-o://9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b" gracePeriod=30
Nov 03 18:04:13.334462 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:13.334422    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"]
Nov 03 18:04:13.335165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:13.334693    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy" containerID="cri-o://097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977" gracePeriod=30
Nov 03 18:04:14.242318 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.240927    4493 generic.go:356] "Generic (PLEG): container finished" podID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerID="4699fe48133688a6fbc9042ec5bf7f7fc5475f7c6a0ebc8a28b0c5e7f390bf6b" exitCode=0
Nov 03 18:04:14.242318 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.241197    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" event={"ID":"3a51d890-33b9-40a4-81c3-d2f54f778891","Type":"ContainerDied","Data":"4699fe48133688a6fbc9042ec5bf7f7fc5475f7c6a0ebc8a28b0c5e7f390bf6b"}
Nov 03 18:04:14.246237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.246047    4493 generic.go:356] "Generic (PLEG): container finished" podID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerID="71496b777fc1bad0521b6c3c2d43445b0120a875ca4c743d7848420fc4d5ac0d" exitCode=0
Nov 03 18:04:14.246237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.246169    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" event={"ID":"5b8341c4-0eb6-489f-b16e-d190ba76201f","Type":"ContainerDied","Data":"71496b777fc1bad0521b6c3c2d43445b0120a875ca4c743d7848420fc4d5ac0d"}
Nov 03 18:04:14.249205 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.249051    4493 generic.go:356] "Generic (PLEG): container finished" podID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerID="a75df992e1f7cca5a57617ea165768fc3cc2bb05488df02d3d91dc5e56e4fe9f" exitCode=0
Nov 03 18:04:14.249205 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.249124    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" event={"ID":"add8c803-d76c-42e5-99a7-e4c41824e10e","Type":"ContainerDied","Data":"a75df992e1f7cca5a57617ea165768fc3cc2bb05488df02d3d91dc5e56e4fe9f"}
Nov 03 18:04:14.318744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.318439    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:04:14.371168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.371142    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:04:14.382244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.382218    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:04:14.418360 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418327    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-envoy\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.418548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418367    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-credential-socket\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.418548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418393    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-envoy\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.418548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418422    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-kube-api-access-gkp95\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-kube-api-access-gkp95\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.418548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418452    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-certs\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.418548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418482    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a51d890-33b9-40a4-81c3-d2f54f778891-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a51d890-33b9-40a4-81c3-d2f54f778891-istiod-ca-cert\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.418548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418539    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-socket\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.418893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418584    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-data\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.418893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418628    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-podinfo\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.418893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418685    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-credential-socket" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.419062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418930    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-data" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.419156 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419132    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a51d890-33b9-40a4-81c3-d2f54f778891-istiod-ca-cert" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:14.419255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.418699    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-envoy\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.419336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419312    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-podinfo\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.419391 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419349    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-credential-socket\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.419443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419404    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-podinfo\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419496    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-token\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-token\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419535    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-socket\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419582    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-data\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419618    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-socket\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419228    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-certs" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419675    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-token\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-token\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419712    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-kube-api-access-h5swr\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-kube-api-access-h5swr\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419631    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-socket" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419788    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-credential-socket\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419832    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-credential-socket" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419841    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-certs\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419956    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-token\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-token\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419980    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-data" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.419989    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-kube-api-access-6995g\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-kube-api-access-6995g\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.420028    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add8c803-d76c-42e5-99a7-e4c41824e10e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add8c803-d76c-42e5-99a7-e4c41824e10e-istiod-ca-cert\") pod \"add8c803-d76c-42e5-99a7-e4c41824e10e\" (UID: \"add8c803-d76c-42e5-99a7-e4c41824e10e\") "
Nov 03 18:04:14.420147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.420122    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8341c4-0eb6-489f-b16e-d190ba76201f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8341c4-0eb6-489f-b16e-d190ba76201f-istiod-ca-cert\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.421054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.420172    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-data\") pod \"5b8341c4-0eb6-489f-b16e-d190ba76201f\" (UID: \"5b8341c4-0eb6-489f-b16e-d190ba76201f\") "
Nov 03 18:04:14.421054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.420284    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-certs\") pod \"3a51d890-33b9-40a4-81c3-d2f54f778891\" (UID: \"3a51d890-33b9-40a4-81c3-d2f54f778891\") "
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421219    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421245    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421259    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a51d890-33b9-40a4-81c3-d2f54f778891-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421273    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421286    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421291    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-envoy" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421299    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421313    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421538    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-certs" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421550    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-certs" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.422007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.421733    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-socket" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.422911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.422062    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-podinfo" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:14.422911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.422396    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add8c803-d76c-42e5-99a7-e4c41824e10e-istiod-ca-cert" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:14.423527 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.423373    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-podinfo" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:14.423927 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.423850    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8341c4-0eb6-489f-b16e-d190ba76201f-istiod-ca-cert" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:14.425587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.425558    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-envoy" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.425690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.425595    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-token" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.425690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.425613    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-podinfo" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:14.425808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.425795    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-data" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.425928 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.425906    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-token" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.426006 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.425967    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-envoy" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.426088 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.426066    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-credential-socket" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.426149 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.426089    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-socket" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:14.426458 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.426439    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-kube-api-access-6995g" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "kube-api-access-6995g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.426670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.426648    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-kube-api-access-h5swr" pod "3a51d890-33b9-40a4-81c3-d2f54f778891" (UID: "3a51d890-33b9-40a4-81c3-d2f54f778891"). InnerVolumeSpecName "kube-api-access-h5swr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.426742 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.426725    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-kube-api-access-gkp95" pod "add8c803-d76c-42e5-99a7-e4c41824e10e" (UID: "add8c803-d76c-42e5-99a7-e4c41824e10e"). InnerVolumeSpecName "kube-api-access-gkp95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.427757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.427738    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-token" pod "5b8341c4-0eb6-489f-b16e-d190ba76201f" (UID: "5b8341c4-0eb6-489f-b16e-d190ba76201f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:14.522967 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.522925    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.522967 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.522960    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6995g\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-kube-api-access-6995g\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.522975    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/add8c803-d76c-42e5-99a7-e4c41824e10e-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.522991    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b8341c4-0eb6-489f-b16e-d190ba76201f-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523005    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523018    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523031    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523044    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523058    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkp95\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-kube-api-access-gkp95\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523070    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523086    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523098    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a51d890-33b9-40a4-81c3-d2f54f778891-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523110    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523124    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b8341c4-0eb6-489f-b16e-d190ba76201f-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523138    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b8341c4-0eb6-489f-b16e-d190ba76201f-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523151    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523164    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/add8c803-d76c-42e5-99a7-e4c41824e10e-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523179    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5swr\" (UniqueName: \"kubernetes.io/projected/3a51d890-33b9-40a4-81c3-d2f54f778891-kube-api-access-h5swr\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523192    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a51d890-33b9-40a4-81c3-d2f54f778891-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:14.523223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:14.523206    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/add8c803-d76c-42e5-99a7-e4c41824e10e-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:15.257604 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.257572    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"
Nov 03 18:04:15.257604 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.257595    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4" event={"ID":"5b8341c4-0eb6-489f-b16e-d190ba76201f","Type":"ContainerDied","Data":"12083be50631365568478fc302eb48ce4575bce03217ed514f34c0814bd5c310"}
Nov 03 18:04:15.257853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.257651    4493 scope.go:122] "RemoveContainer" containerID="71496b777fc1bad0521b6c3c2d43445b0120a875ca4c743d7848420fc4d5ac0d"
Nov 03 18:04:15.259351 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.259330    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"
Nov 03 18:04:15.259484 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.259362    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j" event={"ID":"add8c803-d76c-42e5-99a7-e4c41824e10e","Type":"ContainerDied","Data":"8e6e31cb15e754c83288a9d884808651bc6014c5e1470a67b24790c5c12f85ad"}
Nov 03 18:04:15.261294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.261279    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"
Nov 03 18:04:15.261399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.261311    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj" event={"ID":"3a51d890-33b9-40a4-81c3-d2f54f778891","Type":"ContainerDied","Data":"1ec39e8e79a46213ddf6f690ee24a1e6cce13f54471f95546c7fc9121abbeb56"}
Nov 03 18:04:15.269472 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.269458    4493 scope.go:122] "RemoveContainer" containerID="a75df992e1f7cca5a57617ea165768fc3cc2bb05488df02d3d91dc5e56e4fe9f"
Nov 03 18:04:15.279164 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.279146    4493 scope.go:122] "RemoveContainer" containerID="4699fe48133688a6fbc9042ec5bf7f7fc5475f7c6a0ebc8a28b0c5e7f390bf6b"
Nov 03 18:04:15.481701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.481670    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"]
Nov 03 18:04:15.532524 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.532450    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-c54nj"]
Nov 03 18:04:15.658947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.658911    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"]
Nov 03 18:04:15.703503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.703474    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-cfq9j"]
Nov 03 18:04:15.776944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.776913    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"]
Nov 03 18:04:15.829614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:15.829532    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-56rr4"]
Nov 03 18:04:16.713555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:16.713519    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" path="/var/lib/kubelet/pods/3a51d890-33b9-40a4-81c3-d2f54f778891/volumes"
Nov 03 18:04:16.714060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:16.714046    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" path="/var/lib/kubelet/pods/5b8341c4-0eb6-489f-b16e-d190ba76201f/volumes"
Nov 03 18:04:16.714518 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:16.714505    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" path="/var/lib/kubelet/pods/add8c803-d76c-42e5-99a7-e4c41824e10e/volumes"
Nov 03 18:04:17.607780 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.607715    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 18:04:17.683464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683437    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-certs\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.683658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683479    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-envoy\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.683658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683517    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-socket\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.683658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683574    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-data\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.683658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683623    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-token\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-token\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.683911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683742    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-certs" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:17.683911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683826    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-podinfo\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.683911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683842    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-data" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:17.683911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683864    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-credential-socket\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.684117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683946    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istiod-ca-cert\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.684117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683943    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-socket" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:17.684117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.683999    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-kube-api-access-wdz7b\") pod \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\" (UID: \"75c84f63-1f56-4d39-af09-ee4a3b8f35e1\") "
Nov 03 18:04:17.684117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.684085    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-credential-socket" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:17.684357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.684319    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istiod-ca-cert" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:17.684806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.684746    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.684806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.684773    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.684806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.684786    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.684806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.684799    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.685114 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.684813    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.686263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.686233    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-podinfo" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:17.686422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.686405    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-kube-api-access-wdz7b" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "kube-api-access-wdz7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:17.686548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.686526    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-envoy" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:17.686589 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.686547    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-token" pod "75c84f63-1f56-4d39-af09-ee4a3b8f35e1" (UID: "75c84f63-1f56-4d39-af09-ee4a3b8f35e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:17.786940 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.786906    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.786940 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.786943    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdz7b\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-kube-api-access-wdz7b\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.787375 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.786961    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:17.787375 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:17.786974    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c84f63-1f56-4d39-af09-ee4a3b8f35e1-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.291523 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.291486    4493 generic.go:356] "Generic (PLEG): container finished" podID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerID="a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69" exitCode=0
Nov 03 18:04:18.291688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.291608    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"
Nov 03 18:04:18.291688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.291623    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" event={"ID":"75c84f63-1f56-4d39-af09-ee4a3b8f35e1","Type":"ContainerDied","Data":"a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69"}
Nov 03 18:04:18.291688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.291682    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h" event={"ID":"75c84f63-1f56-4d39-af09-ee4a3b8f35e1","Type":"ContainerDied","Data":"5cbcaf843b46d8cbfb3f7cf08063b861e68b130fa15b2b6088f5ef7cae0abb3b"}
Nov 03 18:04:18.291843 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.291705    4493 scope.go:122] "RemoveContainer" containerID="a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69"
Nov 03 18:04:18.351842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.351817    4493 scope.go:122] "RemoveContainer" containerID="a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69"
Nov 03 18:04:18.352680 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:04:18.352177    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69\": container with ID starting with a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69 not found: ID does not exist" containerID="a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69"
Nov 03 18:04:18.352680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.352210    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69"} err="failed to get container status \"a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69\": rpc error: code = NotFound desc = could not find container \"a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69\": container with ID starting with a9f27ea2a567b8a179ac2d127f938b79941831733084f5279b92b0d127c4bf69 not found: ID does not exist"
Nov 03 18:04:18.538323 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.538278    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"]
Nov 03 18:04:18.589516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.589469    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-zzf5h"]
Nov 03 18:04:18.654469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.651113    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 18:04:18.774973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.774720    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" path="/var/lib/kubelet/pods/75c84f63-1f56-4d39-af09-ee4a3b8f35e1/volumes"
Nov 03 18:04:18.778483 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.778401    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 18:04:18.863661 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863579    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-envoy\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863671    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-certs\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863706    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-credential-socket\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863747    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-certs\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863833    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-data\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863892    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-socket\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863930    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-kube-api-access-lqx2d\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-kube-api-access-lqx2d\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.863958    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istiod-ca-cert\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864005    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-certs" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864055    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-data\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864072    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-certs" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864110    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-podinfo\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864161    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-credential-socket\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864199    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istiod-ca-cert\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864227    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-socket\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864254    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-podinfo\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864279    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-kube-api-access-vtt8f\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-kube-api-access-vtt8f\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864302    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-data" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864356    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-envoy\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864390    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-token\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-token\") pod \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\" (UID: \"11dcf183-0ab9-44f5-a9bc-6989be08e24c\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864392    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istiod-ca-cert" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864425    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-token\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-token\") pod \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\" (UID: \"8b257bb2-2d65-4ada-a050-c5260dcc4dea\") "
Nov 03 18:04:18.864855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864596    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-socket" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.864962    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-credential-socket" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865005    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-socket" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865122    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istiod-ca-cert" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865264    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865282    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865297    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865312    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865327    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865341    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865354    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.865409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865367    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.866021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.865550    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-credential-socket" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.867335 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.867280    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-data" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.868094 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.868058    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-kube-api-access-lqx2d" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "kube-api-access-lqx2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:18.868363 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.868339    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-kube-api-access-vtt8f" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "kube-api-access-vtt8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:18.868596 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.868570    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-podinfo" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:18.869003 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.868968    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-podinfo" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:04:18.869621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.869584    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-token" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:18.870054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.870020    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-token" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:04:18.870813 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.870787    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-envoy" pod "8b257bb2-2d65-4ada-a050-c5260dcc4dea" (UID: "8b257bb2-2d65-4ada-a050-c5260dcc4dea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.873461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.873433    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-envoy" pod "11dcf183-0ab9-44f5-a9bc-6989be08e24c" (UID: "11dcf183-0ab9-44f5-a9bc-6989be08e24c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:04:18.969587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969537    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969576    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969589    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtt8f\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-kube-api-access-vtt8f\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969760    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969771    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11dcf183-0ab9-44f5-a9bc-6989be08e24c-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969807    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969819    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969830    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969842    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqx2d\" (UniqueName: \"kubernetes.io/projected/8b257bb2-2d65-4ada-a050-c5260dcc4dea-kube-api-access-lqx2d\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:18.969968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:18.969854    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b257bb2-2d65-4ada-a050-c5260dcc4dea-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:04:19.300722 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.300686    4493 generic.go:356] "Generic (PLEG): container finished" podID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerID="9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b" exitCode=0
Nov 03 18:04:19.300939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.300739    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"
Nov 03 18:04:19.300939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.300779    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" event={"ID":"11dcf183-0ab9-44f5-a9bc-6989be08e24c","Type":"ContainerDied","Data":"9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b"}
Nov 03 18:04:19.300939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.300835    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j" event={"ID":"11dcf183-0ab9-44f5-a9bc-6989be08e24c","Type":"ContainerDied","Data":"f4e93fca17b68ae62a1157487336b2dc0ef0654ed82c53132d9306b09fa5464f"}
Nov 03 18:04:19.300939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.300857    4493 scope.go:122] "RemoveContainer" containerID="9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b"
Nov 03 18:04:19.303350 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.303329    4493 generic.go:356] "Generic (PLEG): container finished" podID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerID="097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977" exitCode=0
Nov 03 18:04:19.303550 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.303386    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"
Nov 03 18:04:19.303550 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.303410    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" event={"ID":"8b257bb2-2d65-4ada-a050-c5260dcc4dea","Type":"ContainerDied","Data":"097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977"}
Nov 03 18:04:19.303550 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.303450    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8" event={"ID":"8b257bb2-2d65-4ada-a050-c5260dcc4dea","Type":"ContainerDied","Data":"c5131ce47a8dd951efee81405f0b97856ea4decd35abf1f2447bb91566a4a75a"}
Nov 03 18:04:19.313328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.313312    4493 scope.go:122] "RemoveContainer" containerID="9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b"
Nov 03 18:04:19.313629 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:04:19.313611    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b\": container with ID starting with 9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b not found: ID does not exist" containerID="9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b"
Nov 03 18:04:19.313689 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.313637    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b"} err="failed to get container status \"9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b\": rpc error: code = NotFound desc = could not find container \"9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b\": container with ID starting with 9551d03e8ac3ee95613aceb235595ea79a44fd93c1d17f7ba602f1580dbdb94b not found: ID does not exist"
Nov 03 18:04:19.313689 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.313653    4493 scope.go:122] "RemoveContainer" containerID="097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977"
Nov 03 18:04:19.324461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.324438    4493 scope.go:122] "RemoveContainer" containerID="097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977"
Nov 03 18:04:19.324774 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:04:19.324742    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977\": container with ID starting with 097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977 not found: ID does not exist" containerID="097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977"
Nov 03 18:04:19.324894 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.324782    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977"} err="failed to get container status \"097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977\": rpc error: code = NotFound desc = could not find container \"097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977\": container with ID starting with 097b8a26c8f378b9c7579e4c0e8b1516b787ad6ba831bda7a62f03579786e977 not found: ID does not exist"
Nov 03 18:04:19.506599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.506562    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"]
Nov 03 18:04:19.553581 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.553502    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-zzwv8"]
Nov 03 18:04:19.622496 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.622455    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"]
Nov 03 18:04:19.673587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:19.673548    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-gcb2j"]
Nov 03 18:04:20.713452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:20.713424    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" path="/var/lib/kubelet/pods/11dcf183-0ab9-44f5-a9bc-6989be08e24c/volumes"
Nov 03 18:04:20.713904 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:20.713865    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" path="/var/lib/kubelet/pods/8b257bb2-2d65-4ada-a050-c5260dcc4dea/volumes"
Nov 03 18:04:36.993342 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:04:36.993300    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:04:41.703083 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.703051    4493 scope.go:122] "RemoveContainer" containerID="d6bc7a7da3e4637a2efea7c8ec00624231d895d4ece14e39fdaba51c15927bb2"
Nov 03 18:04:41.712908 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.712856    4493 scope.go:122] "RemoveContainer" containerID="d852c519f0a27f237f60cbd934f49975dd2664575fb1a7f4bc6fac63b040a03a"
Nov 03 18:04:41.722097 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.722070    4493 scope.go:122] "RemoveContainer" containerID="e563c6bf27b4990a541c17fd5686fa6e4b41691f366abdea80306a21ea6adb33"
Nov 03 18:04:41.731115 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.731093    4493 scope.go:122] "RemoveContainer" containerID="4d79b04c866ea31a28017c4206d8cd19380456392cb7081d1214876cc1a2cd01"
Nov 03 18:04:41.739955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.739931    4493 scope.go:122] "RemoveContainer" containerID="a3d6e78a1d08f1192f29e87079cac52bba11a07f6126666d636f7902d25ae24d"
Nov 03 18:04:41.749014 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.748994    4493 scope.go:122] "RemoveContainer" containerID="a20054213ee984224b4cc6cd7772ca5a2668c2118074b0489380f3e6e3b3def3"
Nov 03 18:04:41.757383 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.757360    4493 scope.go:122] "RemoveContainer" containerID="29bf9e2e47850fb9c6d910ab836f6efc15ba244ce0f07439f6ebb84b5d8123f8"
Nov 03 18:04:41.766032 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.766013    4493 scope.go:122] "RemoveContainer" containerID="b045706cbd7eccccaa10e87daf893fcfb96e9661872622a6d51ca560bf4092c9"
Nov 03 18:04:41.774793 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.774775    4493 scope.go:122] "RemoveContainer" containerID="1f3c926c39eb91705048ababd90f1f37ad9138495cca608bbb8f649f24dd1ce7"
Nov 03 18:04:41.783766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:04:41.783746    4493 scope.go:122] "RemoveContainer" containerID="16c4cc301fe9e1cdabcb411c546257ae02c8a9bf64393c8fb696650c6887011c"
Nov 03 18:05:06.833436 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.833395    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"]
Nov 03 18:05:06.833978 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.833964    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy"
Nov 03 18:05:06.834026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.833979    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy"
Nov 03 18:05:06.834026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.833990    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7eae40ab-5a22-414f-9792-82b31b272ecc" containerName="revoke-keys"
Nov 03 18:05:06.834026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.833995    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eae40ab-5a22-414f-9792-82b31b272ecc" containerName="revoke-keys"
Nov 03 18:05:06.834026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834005    4493 cpu_manager.go:414] "containerMap: removing container" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy"
Nov 03 18:05:06.834026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834011    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy"
Nov 03 18:05:06.834026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834020    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy"
Nov 03 18:05:06.834026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834024    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834034    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834038    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834054    4493 cpu_manager.go:414] "containerMap: removing container" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834059    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834068    4493 cpu_manager.go:414] "containerMap: removing container" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834072    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834158    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a51d890-33b9-40a4-81c3-d2f54f778891" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834167    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8341c4-0eb6-489f-b16e-d190ba76201f" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834177    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="add8c803-d76c-42e5-99a7-e4c41824e10e" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834183    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dcf183-0ab9-44f5-a9bc-6989be08e24c" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834192    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b257bb2-2d65-4ada-a050-c5260dcc4dea" containerName="istio-proxy"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834197    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eae40ab-5a22-414f-9792-82b31b272ecc" containerName="revoke-keys"
Nov 03 18:05:06.834239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.834202    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c84f63-1f56-4d39-af09-ee4a3b8f35e1" containerName="istio-proxy"
Nov 03 18:05:06.837358 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.837342    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:06.896292 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.896263    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"]
Nov 03 18:05:06.931089 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.931054    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:06.931230 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.931099    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:06.931295 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.931271    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczkp\" (UniqueName: \"kubernetes.io/projected/ceab99bb-e1bf-4339-aafc-139f70a76b69-kube-api-access-qczkp\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:06.942385 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.942360    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:05:06.942385 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.942385    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:05:06.942560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.942414    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:05:06.942560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:06.942453    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:05:06.994237 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:05:06.994204    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:05:07.032439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.032413    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczkp\" (UniqueName: \"kubernetes.io/projected/ceab99bb-e1bf-4339-aafc-139f70a76b69-kube-api-access-qczkp\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:07.032612 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.032597    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:07.032681 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.032620    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:07.033234 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.033217    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:07.035088 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.035071    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:07.098434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.098374    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczkp\" (UniqueName: \"kubernetes.io/projected/ceab99bb-e1bf-4339-aafc-139f70a76b69-kube-api-access-qczkp\") pod \"maas-api-revoke-sub-812e3d9a-nkqmz\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:07.151373 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.151343    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:07.539499 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:05:07.539468    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceab99bb_e1bf_4339_aafc_139f70a76b69.slice/crio-739f1d92d5fc6716c6eac15b8348e9ce1a001abf2c2573429fff4caf46a72751 WatchSource:0}: Error finding container 739f1d92d5fc6716c6eac15b8348e9ce1a001abf2c2573429fff4caf46a72751: Status 404 returned error can't find the container with id 739f1d92d5fc6716c6eac15b8348e9ce1a001abf2c2573429fff4caf46a72751
Nov 03 18:05:07.540024 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.539864    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"]
Nov 03 18:05:07.727575 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:07.727542    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz" event={"ID":"ceab99bb-e1bf-4339-aafc-139f70a76b69","Type":"ContainerStarted","Data":"739f1d92d5fc6716c6eac15b8348e9ce1a001abf2c2573429fff4caf46a72751"}
Nov 03 18:05:08.736076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:08.736043    4493 generic.go:356] "Generic (PLEG): container finished" podID="ceab99bb-e1bf-4339-aafc-139f70a76b69" containerID="4440e7ecaf45e046f9d017d4e9ab194f966fb019b25c06b01e825d6fbcf69356" exitCode=0
Nov 03 18:05:08.736520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:08.736134    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz" event={"ID":"ceab99bb-e1bf-4339-aafc-139f70a76b69","Type":"ContainerDied","Data":"4440e7ecaf45e046f9d017d4e9ab194f966fb019b25c06b01e825d6fbcf69356"}
Nov 03 18:05:09.905828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.905803    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:09.974460 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.974418    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceab99bb-e1bf-4339-aafc-139f70a76b69-kube-api-access-qczkp\" (UniqueName: \"kubernetes.io/projected/ceab99bb-e1bf-4339-aafc-139f70a76b69-kube-api-access-qczkp\") pod \"ceab99bb-e1bf-4339-aafc-139f70a76b69\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") "
Nov 03 18:05:09.974672 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.974481    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-xks-ca\") pod \"ceab99bb-e1bf-4339-aafc-139f70a76b69\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") "
Nov 03 18:05:09.974672 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.974661    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-openshift-ca\") pod \"ceab99bb-e1bf-4339-aafc-139f70a76b69\" (UID: \"ceab99bb-e1bf-4339-aafc-139f70a76b69\") "
Nov 03 18:05:09.975049 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.975024    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-openshift-ca" pod "ceab99bb-e1bf-4339-aafc-139f70a76b69" (UID: "ceab99bb-e1bf-4339-aafc-139f70a76b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:05:09.975251 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.975235    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:09.977009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.976978    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-xks-ca" pod "ceab99bb-e1bf-4339-aafc-139f70a76b69" (UID: "ceab99bb-e1bf-4339-aafc-139f70a76b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:05:09.977113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:09.977091    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceab99bb-e1bf-4339-aafc-139f70a76b69-kube-api-access-qczkp" pod "ceab99bb-e1bf-4339-aafc-139f70a76b69" (UID: "ceab99bb-e1bf-4339-aafc-139f70a76b69"). InnerVolumeSpecName "kube-api-access-qczkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:05:10.076774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:10.076679    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qczkp\" (UniqueName: \"kubernetes.io/projected/ceab99bb-e1bf-4339-aafc-139f70a76b69-kube-api-access-qczkp\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:10.076774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:10.076718    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceab99bb-e1bf-4339-aafc-139f70a76b69-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:05:10.753646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:10.753612    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"
Nov 03 18:05:10.753816 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:10.753676    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz" event={"ID":"ceab99bb-e1bf-4339-aafc-139f70a76b69","Type":"ContainerDied","Data":"739f1d92d5fc6716c6eac15b8348e9ce1a001abf2c2573429fff4caf46a72751"}
Nov 03 18:05:10.753816 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:10.753733    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="739f1d92d5fc6716c6eac15b8348e9ce1a001abf2c2573429fff4caf46a72751"
Nov 03 18:05:36.993581 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:05:36.993545    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:05:41.888910 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:41.888321    4493 scope.go:122] "RemoveContainer" containerID="32eef1ed380dd5201922204eab72d86d990084cdd888c2f2dea0fcda95f6a9f1"
Nov 03 18:05:41.920978 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:05:41.920863    4493 scope.go:122] "RemoveContainer" containerID="d1942bdc2f7abdf38fc3c773bf6f0194e10837cacb902cb6d62502043fee8e21"
Nov 03 18:06:06.995001 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:06:06.994906    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:06:11.562979 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.562944    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"]
Nov 03 18:06:11.563543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.563528    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ceab99bb-e1bf-4339-aafc-139f70a76b69" containerName="revoke-keys"
Nov 03 18:06:11.563591 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.563544    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceab99bb-e1bf-4339-aafc-139f70a76b69" containerName="revoke-keys"
Nov 03 18:06:11.563657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.563648    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceab99bb-e1bf-4339-aafc-139f70a76b69" containerName="revoke-keys"
Nov 03 18:06:11.566730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.566712    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601449 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601418    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601593 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601455    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b33821-8e02-409a-a925-380e48ea810c-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601593 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601483    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9gxx\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-kube-api-access-p9gxx\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601682 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601619    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601719    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601779 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601751    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601862    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.601978 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601965    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.602024 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.601986    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b33821-8e02-409a-a925-380e48ea810c-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.620812 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.620791    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ad914d-openshift-default-dockercfg-9xgcc\""
Nov 03 18:06:11.704207 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704173    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704219    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b33821-8e02-409a-a925-380e48ea810c-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704356    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704394    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b33821-8e02-409a-a925-380e48ea810c-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704637 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704433    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gxx\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-kube-api-access-p9gxx\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704637 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704551    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704756 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704645    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704756 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704670    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704746    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.704892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.704763    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.710043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.705190    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.710043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.706138    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b33821-8e02-409a-a925-380e48ea810c-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.710043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.706381    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.710043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.706619    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.710043 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.708617    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b33821-8e02-409a-a925-380e48ea810c-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.713035 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.713005    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.769131 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.769096    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.769288 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.769224    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gxx\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-kube-api-access-p9gxx\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:11.785821 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.785778    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"]
Nov 03 18:06:11.880401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:11.880321    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:12.326590 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:12.326563    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"]
Nov 03 18:06:12.326719 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:06:12.326660    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b33821_8e02_409a_a925_380e48ea810c.slice/crio-7d8a874acff4a6c7fc9a22e5a2c3469cd407899023db86556a387055d7552df0 WatchSource:0}: Error finding container 7d8a874acff4a6c7fc9a22e5a2c3469cd407899023db86556a387055d7552df0: Status 404 returned error can't find the container with id 7d8a874acff4a6c7fc9a22e5a2c3469cd407899023db86556a387055d7552df0
Nov 03 18:06:12.329579 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:12.329547    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:12.329674 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:12.329628    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:12.329713 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:12.329679    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:06:12.379781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:12.379752    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" event={"ID":"94b33821-8e02-409a-a925-380e48ea810c","Type":"ContainerStarted","Data":"7d8a874acff4a6c7fc9a22e5a2c3469cd407899023db86556a387055d7552df0"}
Nov 03 18:06:13.387909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:13.387843    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" event={"ID":"94b33821-8e02-409a-a925-380e48ea810c","Type":"ContainerStarted","Data":"aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18"}
Nov 03 18:06:13.564419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:13.564364    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podStartSLOduration=2.56434734 podStartE2EDuration="2.56434734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.562066224 +0000 UTC m=+1953.021237222" watchObservedRunningTime="2026-11-03 18:06:13.56434734 +0000 UTC m=+1953.023518323"
Nov 03 18:06:13.881319 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:13.881287    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:13.883081 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:13.883059    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 18:06:13.883194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:13.883106    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 18:06:14.881095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:14.881056    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 18:06:14.881589 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:14.881119    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 18:06:15.880847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:15.880809    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 18:06:15.881235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:15.881198    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 18:06:16.880864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:16.880822    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 18:06:16.881076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:16.880915    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 18:06:17.881536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:17.881507    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 18:06:17.881937 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:17.881557    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 18:06:18.881047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:18.881009    4493 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 18:06:18.881223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:18.881072    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 18:06:19.887487 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:19.887458    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:19.887925 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:19.887792    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:19.888598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:19.888577    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:06:22.058503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:22.058471    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"]
Nov 03 18:06:22.110957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:22.110865    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c1b7f95-rtqjl"]
Nov 03 18:06:22.713659 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:22.713618    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c74362-a1e5-4f87-81d2-3f6560edafd1" path="/var/lib/kubelet/pods/71c74362-a1e5-4f87-81d2-3f6560edafd1/volumes"
Nov 03 18:06:28.104458 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:28.104427    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"]
Nov 03 18:06:28.104920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:28.104623    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy" containerID="cri-o://33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2" gracePeriod=30
Nov 03 18:06:28.970316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:28.970274    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"]
Nov 03 18:06:28.970645 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:28.970609    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" containerID="cri-o://0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c" gracePeriod=30
Nov 03 18:06:30.940284 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:30.940250    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"]
Nov 03 18:06:30.940677 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:30.940450    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" containerID="cri-o://be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b" gracePeriod=30
Nov 03 18:06:30.960224 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:30.960198    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"]
Nov 03 18:06:30.960416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:30.960382    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" containerID="cri-o://ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567" gracePeriod=30
Nov 03 18:06:32.179588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:32.179551    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:06:32.179992 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:32.179622    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:06:33.383095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.383037    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 18:06:33.461603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.461573    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-socket\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.461755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.461623    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf94c4a-b570-4670-942c-8e981619c397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf94c4a-b570-4670-942c-8e981619c397-istiod-ca-cert\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.461840 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.461819    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-socket" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:33.461914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.461857    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-envoy\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.461969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.461910    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-data\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.461969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.461955    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-certs\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.462074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.461991    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-istio-token\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-istio-token\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.462074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462054    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cf94c4a-b570-4670-942c-8e981619c397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf94c4a-b570-4670-942c-8e981619c397-istio-podinfo\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.462185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462083    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf94c4a-b570-4670-942c-8e981619c397-istiod-ca-cert" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:33.462185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462109    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-credential-socket\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.462185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462161    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-kube-api-access-sb26w\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-kube-api-access-sb26w\") pod \"8cf94c4a-b570-4670-942c-8e981619c397\" (UID: \"8cf94c4a-b570-4670-942c-8e981619c397\") "
Nov 03 18:06:33.462339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462248    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-data" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:33.462339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462227    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-certs" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:33.462339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462310    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-credential-socket" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:33.462812 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462785    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.462964 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462817    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.462964 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462832    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.462964 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462848    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.462964 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.462862    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf94c4a-b570-4670-942c-8e981619c397-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.464284 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.464258    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-envoy" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:33.464399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.464363    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cf94c4a-b570-4670-942c-8e981619c397-istio-podinfo" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:06:33.464456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.464436    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-istio-token" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:33.464608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.464586    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-kube-api-access-sb26w" pod "8cf94c4a-b570-4670-942c-8e981619c397" (UID: "8cf94c4a-b570-4670-942c-8e981619c397"). InnerVolumeSpecName "kube-api-access-sb26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:33.536977 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.536944    4493 generic.go:356] "Generic (PLEG): container finished" podID="8cf94c4a-b570-4670-942c-8e981619c397" containerID="33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2" exitCode=0
Nov 03 18:06:33.537126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.537003    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"
Nov 03 18:06:33.537126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.537037    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" event={"ID":"8cf94c4a-b570-4670-942c-8e981619c397","Type":"ContainerDied","Data":"33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2"}
Nov 03 18:06:33.537126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.537097    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd" event={"ID":"8cf94c4a-b570-4670-942c-8e981619c397","Type":"ContainerDied","Data":"7f73dbd8d4596102f7e9580b00decce35a8b3bce7e5bccb9571be7b86153bf57"}
Nov 03 18:06:33.537269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.537127    4493 scope.go:122] "RemoveContainer" containerID="33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2"
Nov 03 18:06:33.547773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.547754    4493 scope.go:122] "RemoveContainer" containerID="33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2"
Nov 03 18:06:33.548112 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:06:33.548093    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2\": container with ID starting with 33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2 not found: ID does not exist" containerID="33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2"
Nov 03 18:06:33.548171 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.548121    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2"} err="failed to get container status \"33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2\": rpc error: code = NotFound desc = could not find container \"33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2\": container with ID starting with 33e5f1dc6793b2c031fe4d7db81b4e51b5de5963d49bdc41aa5c7107ee2af5f2 not found: ID does not exist"
Nov 03 18:06:33.564064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.564036    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf94c4a-b570-4670-942c-8e981619c397-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.564064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.564066    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.564256 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.564082    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf94c4a-b570-4670-942c-8e981619c397-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.564256 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.564095    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb26w\" (UniqueName: \"kubernetes.io/projected/8cf94c4a-b570-4670-942c-8e981619c397-kube-api-access-sb26w\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:33.681562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.681483    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"]
Nov 03 18:06:33.732009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.731980    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-t6cgd"]
Nov 03 18:06:33.795699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.795666    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:06:33.795865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:33.795729    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:06:34.261901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.261843    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 18:06:34.375489 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375454    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-socket\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.375684 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375502    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-credential-socket\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.375684 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375577    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-envoy\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.375684 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375672    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-kube-api-access-8vskl\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-kube-api-access-8vskl\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.375844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375724    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-podinfo\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.375844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375787    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-socket" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:34.375982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375914    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-token\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-token\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.376037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.375911    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-credential-socket" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:34.376095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.376034    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-data\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.376241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.376220    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-certs\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.376371 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.376272    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be37b81-fe72-47b6-abfd-fb0f6933beac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be37b81-fe72-47b6-abfd-fb0f6933beac-istiod-ca-cert\") pod \"5be37b81-fe72-47b6-abfd-fb0f6933beac\" (UID: \"5be37b81-fe72-47b6-abfd-fb0f6933beac\") "
Nov 03 18:06:34.376539 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.376510    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-certs" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:34.376646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.376621    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-data" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:34.376886 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.376847    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be37b81-fe72-47b6-abfd-fb0f6933beac-istiod-ca-cert" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:34.377376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.377348    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.377499 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.377388    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.377499 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.377406    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be37b81-fe72-47b6-abfd-fb0f6933beac-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.377499 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.377421    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.377499 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.377435    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.380265 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.379458    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-podinfo" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:06:34.380265 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.380227    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-envoy" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:34.380572 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.380544    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-token" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:34.380681 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.380617    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-kube-api-access-8vskl" pod "5be37b81-fe72-47b6-abfd-fb0f6933beac" (UID: "5be37b81-fe72-47b6-abfd-fb0f6933beac"). InnerVolumeSpecName "kube-api-access-8vskl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:34.478701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.478664    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.478701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.478694    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.478701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.478705    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vskl\" (UniqueName: \"kubernetes.io/projected/5be37b81-fe72-47b6-abfd-fb0f6933beac-kube-api-access-8vskl\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.479167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.478716    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be37b81-fe72-47b6-abfd-fb0f6933beac-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:34.544646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.544548    4493 generic.go:356] "Generic (PLEG): container finished" podID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerID="0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c" exitCode=0
Nov 03 18:06:34.544646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.544602    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"
Nov 03 18:06:34.544646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.544639    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" event={"ID":"5be37b81-fe72-47b6-abfd-fb0f6933beac","Type":"ContainerDied","Data":"0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c"}
Nov 03 18:06:34.544970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.544690    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc" event={"ID":"5be37b81-fe72-47b6-abfd-fb0f6933beac","Type":"ContainerDied","Data":"2915fdbfd51145ce5d2389653566de31399dd68fb5716a0ddef314dc196b3ec9"}
Nov 03 18:06:34.544970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.544718    4493 scope.go:122] "RemoveContainer" containerID="0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c"
Nov 03 18:06:34.559703 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.559320    4493 scope.go:122] "RemoveContainer" containerID="0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c"
Nov 03 18:06:34.559776 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:06:34.559692    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c\": container with ID starting with 0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c not found: ID does not exist" containerID="0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c"
Nov 03 18:06:34.559776 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.559726    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c"} err="failed to get container status \"0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c\": rpc error: code = NotFound desc = could not find container \"0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c\": container with ID starting with 0c3b0af4c5ed7c1c94a2ff54bdbd8fbf13e8f0abfe5fddcae0c985221535568c not found: ID does not exist"
Nov 03 18:06:34.716547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.716513    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf94c4a-b570-4670-942c-8e981619c397" path="/var/lib/kubelet/pods/8cf94c4a-b570-4670-942c-8e981619c397/volumes"
Nov 03 18:06:34.718247 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.718227    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"]
Nov 03 18:06:34.780219 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:34.780170    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-h79lc"]
Nov 03 18:06:35.727110 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:35.727077    4493 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:06:35.727561 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:35.727133    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:06:36.442947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.441363    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 18:06:36.467834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.467805    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 18:06:36.563181 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.563094    4493 generic.go:356] "Generic (PLEG): container finished" podID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerID="ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567" exitCode=0
Nov 03 18:06:36.563181 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.563144    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"
Nov 03 18:06:36.563412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.563181    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" event={"ID":"049c9fa4-86ff-4958-b699-1156e7df1a3a","Type":"ContainerDied","Data":"ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567"}
Nov 03 18:06:36.563412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.563258    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj" event={"ID":"049c9fa4-86ff-4958-b699-1156e7df1a3a","Type":"ContainerDied","Data":"349056785b4e26a08c5c42f3bb0c1fdde8d5979dc3b33592b61020a144daaa7c"}
Nov 03 18:06:36.563412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.563287    4493 scope.go:122] "RemoveContainer" containerID="ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567"
Nov 03 18:06:36.564934 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.564813    4493 generic.go:356] "Generic (PLEG): container finished" podID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerID="be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b" exitCode=0
Nov 03 18:06:36.564934 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.564864    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"
Nov 03 18:06:36.565112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.564862    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" event={"ID":"1c7908bb-be3a-473a-9179-e511506c7eb4","Type":"ContainerDied","Data":"be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b"}
Nov 03 18:06:36.565112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.565007    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp" event={"ID":"1c7908bb-be3a-473a-9179-e511506c7eb4","Type":"ContainerDied","Data":"fc72f70f3de67a04b964539ee68a388adea136d28c8dd6526af94a0cfd29a03d"}
Nov 03 18:06:36.573985 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.573965    4493 scope.go:122] "RemoveContainer" containerID="ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567"
Nov 03 18:06:36.574315 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:06:36.574296    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567\": container with ID starting with ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567 not found: ID does not exist" containerID="ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567"
Nov 03 18:06:36.574404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.574328    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567"} err="failed to get container status \"ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567\": rpc error: code = NotFound desc = could not find container \"ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567\": container with ID starting with ae93a5cfc3cb4cb1e4e5b240e881891bec552104299fcbb99f9fa7e9c73d0567 not found: ID does not exist"
Nov 03 18:06:36.574404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.574352    4493 scope.go:122] "RemoveContainer" containerID="be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b"
Nov 03 18:06:36.584306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.584280    4493 scope.go:122] "RemoveContainer" containerID="be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b"
Nov 03 18:06:36.584649 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:06:36.584607    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b\": container with ID starting with be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b not found: ID does not exist" containerID="be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b"
Nov 03 18:06:36.584742 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.584655    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b"} err="failed to get container status \"be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b\": rpc error: code = NotFound desc = could not find container \"be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b\": container with ID starting with be90fb61960171411d875d6ea8b9c31ed16445e71dbf91bf98e12cbd37f8951b not found: ID does not exist"
Nov 03 18:06:36.601814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.601783    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-certs\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.601944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.601839    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-data\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.602011 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.601991    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-envoy\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.602060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602025    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-envoy\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.602109 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602063    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-kube-api-access-fqj9s\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.602157 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602110    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-token\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-token\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.602157 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602026    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-certs" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.602157 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602145    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-certs\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.602157 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602115    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-data" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.602368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602220    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-podinfo\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.602368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602249    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-kube-api-access-l6v8n\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-kube-api-access-l6v8n\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.602368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602307    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-data\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602601    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-data" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602718    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-podinfo\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602789    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-certs" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602797    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-credential-socket\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602886    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-socket\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602924    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-socket\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602979    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7908bb-be3a-473a-9179-e511506c7eb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7908bb-be3a-473a-9179-e511506c7eb4-istiod-ca-cert\") pod \"1c7908bb-be3a-473a-9179-e511506c7eb4\" (UID: \"1c7908bb-be3a-473a-9179-e511506c7eb4\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.602985    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-credential-socket" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.603034    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-token\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-token\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.603146    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-credential-socket\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.603197    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049c9fa4-86ff-4958-b699-1156e7df1a3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c9fa4-86ff-4958-b699-1156e7df1a3a-istiod-ca-cert\") pod \"049c9fa4-86ff-4958-b699-1156e7df1a3a\" (UID: \"049c9fa4-86ff-4958-b699-1156e7df1a3a\") "
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.603735    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7908bb-be3a-473a-9179-e511506c7eb4-istiod-ca-cert" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604130    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-socket" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.604327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604302    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-socket" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604357    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604379    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604395    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604411    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c7908bb-be3a-473a-9179-e511506c7eb4-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604426    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604439    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604452    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.604434    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-credential-socket" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.605275    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-kube-api-access-fqj9s" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "kube-api-access-fqj9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.605355    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-envoy" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.605652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.605528    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049c9fa4-86ff-4958-b699-1156e7df1a3a-istiod-ca-cert" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:36.606262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.606135    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-envoy" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:36.606262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.606202    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-podinfo" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:06:36.606439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.606416    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-token" pod "1c7908bb-be3a-473a-9179-e511506c7eb4" (UID: "1c7908bb-be3a-473a-9179-e511506c7eb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:36.606523 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.606508    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-token" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:36.606898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.606847    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-podinfo" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:06:36.606953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.606919    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-kube-api-access-l6v8n" pod "049c9fa4-86ff-4958-b699-1156e7df1a3a" (UID: "049c9fa4-86ff-4958-b699-1156e7df1a3a"). InnerVolumeSpecName "kube-api-access-l6v8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:36.705775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705742    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.705775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705774    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.705775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705784    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705793    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705803    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049c9fa4-86ff-4958-b699-1156e7df1a3a-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705814    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705822    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049c9fa4-86ff-4958-b699-1156e7df1a3a-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705831    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-kube-api-access-fqj9s\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705841    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705849    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c7908bb-be3a-473a-9179-e511506c7eb4-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.706042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.705857    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6v8n\" (UniqueName: \"kubernetes.io/projected/049c9fa4-86ff-4958-b699-1156e7df1a3a-kube-api-access-l6v8n\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:36.715403 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.715371    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" path="/var/lib/kubelet/pods/5be37b81-fe72-47b6-abfd-fb0f6933beac/volumes"
Nov 03 18:06:36.987620 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:36.987588    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"]
Nov 03 18:06:37.014626 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:06:37.014547    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:06:37.037888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:37.037835    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-p57tj"]
Nov 03 18:06:37.143463 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:37.140042    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"]
Nov 03 18:06:37.146612 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:37.146582    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-2p2sp"]
Nov 03 18:06:38.714227 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:38.714196    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" path="/var/lib/kubelet/pods/049c9fa4-86ff-4958-b699-1156e7df1a3a/volumes"
Nov 03 18:06:38.714776 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:38.714761    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" path="/var/lib/kubelet/pods/1c7908bb-be3a-473a-9179-e511506c7eb4/volumes"
Nov 03 18:06:41.962472 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:41.962439    4493 scope.go:122] "RemoveContainer" containerID="2f4fde627bb9ed96031d5ea09c77ad9559d301eda64e484e424e36e71e3a2cfa"
Nov 03 18:06:41.982187 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:41.976033    4493 scope.go:122] "RemoveContainer" containerID="16823a8bb198fff790efb5ef2c4d56cd5647c6b3cd12705c2a4e1f7a371764ad"
Nov 03 18:06:43.372489 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:43.372457    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"]
Nov 03 18:06:43.372938 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:43.372686    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" containerID="cri-o://8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a" gracePeriod=30
Nov 03 18:06:45.070039 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:45.070004    4493 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:06:45.070442 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:45.070063    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:06:48.540037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.540001    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"]
Nov 03 18:06:48.541864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.541833    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy"
Nov 03 18:06:48.542048 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.542033    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy"
Nov 03 18:06:48.542143 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.542133    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy"
Nov 03 18:06:48.542223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.542212    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy"
Nov 03 18:06:48.542392 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.542380    4493 cpu_manager.go:414] "containerMap: removing container" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy"
Nov 03 18:06:48.542510 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.542499    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy"
Nov 03 18:06:48.542622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.542612    4493 cpu_manager.go:414] "containerMap: removing container" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy"
Nov 03 18:06:48.542713 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.542703    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy"
Nov 03 18:06:48.543222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.543189    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7908bb-be3a-473a-9179-e511506c7eb4" containerName="istio-proxy"
Nov 03 18:06:48.543365 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.543354    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf94c4a-b570-4670-942c-8e981619c397" containerName="istio-proxy"
Nov 03 18:06:48.543457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.543447    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be37b81-fe72-47b6-abfd-fb0f6933beac" containerName="istio-proxy"
Nov 03 18:06:48.543530 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.543521    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="049c9fa4-86ff-4958-b699-1156e7df1a3a" containerName="istio-proxy"
Nov 03 18:06:48.548262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.548238    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.603636 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.603600    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"]
Nov 03 18:06:48.616315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.616283    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.616472 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.616413    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.616526 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.616510    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmwd2\" (UniqueName: \"kubernetes.io/projected/b702075c-3db6-4c76-ba14-932a63ba15de-kube-api-access-tmwd2\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.648505 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.648480    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:06:48.648723 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.648706    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:06:48.648781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.648712    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:06:48.655104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.655083    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:06:48.660614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.660595    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 18:06:48.717434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717397    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-kube-api-access-ls4x9\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-kube-api-access-ls4x9\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.717562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717443    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-credential-socket\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.717562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717514    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-socket\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.717562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717540    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-data\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.717726 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717590    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-envoy\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.717781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717736    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-certs\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.717844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717787    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-credential-socket" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:48.717844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717803    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-socket" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:48.717999 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717864    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-token\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-token\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.717999 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.717949    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-podinfo\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.718110 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.718007    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istiod-ca-cert\") pod \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\" (UID: \"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3\") "
Nov 03 18:06:48.718110 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.718044    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-certs" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:48.718252 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.718223    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-data" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:48.718474 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.718449    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istiod-ca-cert" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:48.718619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.718458    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.719177 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.718863    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmwd2\" (UniqueName: \"kubernetes.io/projected/b702075c-3db6-4c76-ba14-932a63ba15de-kube-api-access-tmwd2\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.719435 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.719413    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.719560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.719543    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.719646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.719565    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.719646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.719580    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.719646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.719596    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.719646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.719609    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.720450 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.720428    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.721034 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.721010    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-kube-api-access-ls4x9" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "kube-api-access-ls4x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:48.721034 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.721020    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-envoy" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:06:48.721034 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.721023    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-podinfo" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:06:48.721215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.721044    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-token" pod "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" (UID: "d63c2ba1-ba7a-4fcd-9624-a929d97a50e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:48.721506 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.721486    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.726699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.726676    4493 generic.go:356] "Generic (PLEG): container finished" podID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerID="8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a" exitCode=0
Nov 03 18:06:48.726790 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.726723    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" event={"ID":"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3","Type":"ContainerDied","Data":"8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a"}
Nov 03 18:06:48.726790 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.726735    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"
Nov 03 18:06:48.726790 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.726765    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt" event={"ID":"d63c2ba1-ba7a-4fcd-9624-a929d97a50e3","Type":"ContainerDied","Data":"836dcae4c26d7ecb8bb42146b6821747ae6747dd2f0f4122ca8a5b229facf89b"}
Nov 03 18:06:48.726949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.726794    4493 scope.go:122] "RemoveContainer" containerID="8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a"
Nov 03 18:06:48.752599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.752583    4493 scope.go:122] "RemoveContainer" containerID="8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a"
Nov 03 18:06:48.752933 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:06:48.752908    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a\": container with ID starting with 8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a not found: ID does not exist" containerID="8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a"
Nov 03 18:06:48.753004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.752946    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a"} err="failed to get container status \"8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a\": rpc error: code = NotFound desc = could not find container \"8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a\": container with ID starting with 8d09231602e77770c97feb59c66ae61efdd8dc545edf46c47e19b37c6be6e89a not found: ID does not exist"
Nov 03 18:06:48.782352 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.782323    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmwd2\" (UniqueName: \"kubernetes.io/projected/b702075c-3db6-4c76-ba14-932a63ba15de-kube-api-access-tmwd2\") pod \"maas-api-revoke-sub-820729f2-f78l5\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.820759 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.820728    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.820759 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.820751    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.820931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.820768    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls4x9\" (UniqueName: \"kubernetes.io/projected/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-kube-api-access-ls4x9\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.820931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.820779    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:48.867301 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.867271    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:48.956261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:48.954382    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"]
Nov 03 18:06:49.004454 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:49.004413    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-rwdjt"]
Nov 03 18:06:49.274010 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:49.273972    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"]
Nov 03 18:06:49.275172 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:06:49.275140    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb702075c_3db6_4c76_ba14_932a63ba15de.slice/crio-c9c89a3f7850caff8886722fee895eb7539416fe3ee7f48b6210313ee4c82071 WatchSource:0}: Error finding container c9c89a3f7850caff8886722fee895eb7539416fe3ee7f48b6210313ee4c82071: Status 404 returned error can't find the container with id c9c89a3f7850caff8886722fee895eb7539416fe3ee7f48b6210313ee4c82071
Nov 03 18:06:49.737867 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:49.737828    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5" event={"ID":"b702075c-3db6-4c76-ba14-932a63ba15de","Type":"ContainerStarted","Data":"589513ec1bcb91f2d3ceedce73c022c32d94091f4912c2e0dd114b617828a6b3"}
Nov 03 18:06:49.738279 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:49.737918    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5" event={"ID":"b702075c-3db6-4c76-ba14-932a63ba15de","Type":"ContainerStarted","Data":"c9c89a3f7850caff8886722fee895eb7539416fe3ee7f48b6210313ee4c82071"}
Nov 03 18:06:49.860649 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:49.860548    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5" podStartSLOduration=1.860533158 podStartE2EDuration="1.860533158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:49.858644567 +0000 UTC m=+1989.317815589" watchObservedRunningTime="2026-11-03 18:06:49.860533158 +0000 UTC m=+1989.319704145"
Nov 03 18:06:50.713610 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:50.713576    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" path="/var/lib/kubelet/pods/d63c2ba1-ba7a-4fcd-9624-a929d97a50e3/volumes"
Nov 03 18:06:50.745566 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:50.745537    4493 generic.go:356] "Generic (PLEG): container finished" podID="b702075c-3db6-4c76-ba14-932a63ba15de" containerID="589513ec1bcb91f2d3ceedce73c022c32d94091f4912c2e0dd114b617828a6b3" exitCode=0
Nov 03 18:06:50.745974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:50.745623    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5" event={"ID":"b702075c-3db6-4c76-ba14-932a63ba15de","Type":"ContainerDied","Data":"589513ec1bcb91f2d3ceedce73c022c32d94091f4912c2e0dd114b617828a6b3"}
Nov 03 18:06:51.920809 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:51.920780    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:52.076411 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.076330    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-xks-ca\") pod \"b702075c-3db6-4c76-ba14-932a63ba15de\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") "
Nov 03 18:06:52.076559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.076521    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-openshift-ca\") pod \"b702075c-3db6-4c76-ba14-932a63ba15de\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") "
Nov 03 18:06:52.076630 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.076584    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b702075c-3db6-4c76-ba14-932a63ba15de-kube-api-access-tmwd2\" (UniqueName: \"kubernetes.io/projected/b702075c-3db6-4c76-ba14-932a63ba15de-kube-api-access-tmwd2\") pod \"b702075c-3db6-4c76-ba14-932a63ba15de\" (UID: \"b702075c-3db6-4c76-ba14-932a63ba15de\") "
Nov 03 18:06:52.076972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.076940    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-openshift-ca" pod "b702075c-3db6-4c76-ba14-932a63ba15de" (UID: "b702075c-3db6-4c76-ba14-932a63ba15de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:06:52.077415 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.077271    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:52.078765 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.078727    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-xks-ca" pod "b702075c-3db6-4c76-ba14-932a63ba15de" (UID: "b702075c-3db6-4c76-ba14-932a63ba15de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:06:52.078920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.078866    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b702075c-3db6-4c76-ba14-932a63ba15de-kube-api-access-tmwd2" pod "b702075c-3db6-4c76-ba14-932a63ba15de" (UID: "b702075c-3db6-4c76-ba14-932a63ba15de"). InnerVolumeSpecName "kube-api-access-tmwd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:06:52.178669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.178624    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmwd2\" (UniqueName: \"kubernetes.io/projected/b702075c-3db6-4c76-ba14-932a63ba15de-kube-api-access-tmwd2\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:52.178669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.178664    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b702075c-3db6-4c76-ba14-932a63ba15de-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:06:52.758447 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.758422    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"
Nov 03 18:06:52.758599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.758456    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5" event={"ID":"b702075c-3db6-4c76-ba14-932a63ba15de","Type":"ContainerDied","Data":"c9c89a3f7850caff8886722fee895eb7539416fe3ee7f48b6210313ee4c82071"}
Nov 03 18:06:52.758599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:52.758501    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c89a3f7850caff8886722fee895eb7539416fe3ee7f48b6210313ee4c82071"
Nov 03 18:06:54.061176 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:54.061147    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"]
Nov 03 18:06:54.112673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:54.112628    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a0042ef-zlf6g"]
Nov 03 18:06:54.714478 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:06:54.714449    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfbbdfce-16d4-485a-9429-148fc050a9e6" path="/var/lib/kubelet/pods/cfbbdfce-16d4-485a-9429-148fc050a9e6/volumes"
Nov 03 18:07:06.992423 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:07:06.992387    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:07:29.345345 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.345261    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"]
Nov 03 18:07:29.345842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.345816    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b702075c-3db6-4c76-ba14-932a63ba15de" containerName="revoke-keys"
Nov 03 18:07:29.345842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.345828    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b702075c-3db6-4c76-ba14-932a63ba15de" containerName="revoke-keys"
Nov 03 18:07:29.345842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.345841    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy"
Nov 03 18:07:29.345979 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.345846    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy"
Nov 03 18:07:29.345979 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.345964    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b702075c-3db6-4c76-ba14-932a63ba15de" containerName="revoke-keys"
Nov 03 18:07:29.345979 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.345975    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63c2ba1-ba7a-4fcd-9624-a929d97a50e3" containerName="istio-proxy"
Nov 03 18:07:29.349348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.349332    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.409543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.409510    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"]
Nov 03 18:07:29.432069 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.432030    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.432235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.432113    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.432328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.432300    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jvc\" (UniqueName: \"kubernetes.io/projected/276f2e0b-545d-416c-b629-9b028aabeeb5-kube-api-access-b6jvc\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.455401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.455369    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:07:29.455401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.455371    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:07:29.455401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.455374    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:07:29.455617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.455416    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:07:29.534006 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.533976    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.534006 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.534007    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.534263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.534089    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jvc\" (UniqueName: \"kubernetes.io/projected/276f2e0b-545d-416c-b629-9b028aabeeb5-kube-api-access-b6jvc\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.534700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.534678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.536529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.536507    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.597805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.597740    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jvc\" (UniqueName: \"kubernetes.io/projected/276f2e0b-545d-416c-b629-9b028aabeeb5-kube-api-access-b6jvc\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:29.663766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:29.663738    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:30.045676 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:30.045643    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"]
Nov 03 18:07:30.047919 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:07:30.047887    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276f2e0b_545d_416c_b629_9b028aabeeb5.slice/crio-b53f66bdfbebd8bdb692db54cedcd3caeefc8b18cde7f26cdd7b356ec90ea1e9 WatchSource:0}: Error finding container b53f66bdfbebd8bdb692db54cedcd3caeefc8b18cde7f26cdd7b356ec90ea1e9: Status 404 returned error can't find the container with id b53f66bdfbebd8bdb692db54cedcd3caeefc8b18cde7f26cdd7b356ec90ea1e9
Nov 03 18:07:31.044356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:31.044318    4493 generic.go:356] "Generic (PLEG): container finished" podID="276f2e0b-545d-416c-b629-9b028aabeeb5" containerID="d444e45d862eb850f763646a7a7c102685f811a34ca4d681ec435eaeafb8c19f" exitCode=0
Nov 03 18:07:31.044810 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:31.044408    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5" event={"ID":"276f2e0b-545d-416c-b629-9b028aabeeb5","Type":"ContainerDied","Data":"d444e45d862eb850f763646a7a7c102685f811a34ca4d681ec435eaeafb8c19f"}
Nov 03 18:07:31.044810 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:31.044461    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5" event={"ID":"276f2e0b-545d-416c-b629-9b028aabeeb5","Type":"ContainerStarted","Data":"b53f66bdfbebd8bdb692db54cedcd3caeefc8b18cde7f26cdd7b356ec90ea1e9"}
Nov 03 18:07:32.226132 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.226105    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:32.279096 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.279046    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-xks-ca\") pod \"276f2e0b-545d-416c-b629-9b028aabeeb5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") "
Nov 03 18:07:32.279286 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.279150    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-openshift-ca\") pod \"276f2e0b-545d-416c-b629-9b028aabeeb5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") "
Nov 03 18:07:32.279286 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.279215    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276f2e0b-545d-416c-b629-9b028aabeeb5-kube-api-access-b6jvc\" (UniqueName: \"kubernetes.io/projected/276f2e0b-545d-416c-b629-9b028aabeeb5-kube-api-access-b6jvc\") pod \"276f2e0b-545d-416c-b629-9b028aabeeb5\" (UID: \"276f2e0b-545d-416c-b629-9b028aabeeb5\") "
Nov 03 18:07:32.279522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.279492    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-openshift-ca" pod "276f2e0b-545d-416c-b629-9b028aabeeb5" (UID: "276f2e0b-545d-416c-b629-9b028aabeeb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:07:32.279818 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.279794    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:32.281399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.281370    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-xks-ca" pod "276f2e0b-545d-416c-b629-9b028aabeeb5" (UID: "276f2e0b-545d-416c-b629-9b028aabeeb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:07:32.281534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.281516    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276f2e0b-545d-416c-b629-9b028aabeeb5-kube-api-access-b6jvc" pod "276f2e0b-545d-416c-b629-9b028aabeeb5" (UID: "276f2e0b-545d-416c-b629-9b028aabeeb5"). InnerVolumeSpecName "kube-api-access-b6jvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:32.381729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.381640    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jvc\" (UniqueName: \"kubernetes.io/projected/276f2e0b-545d-416c-b629-9b028aabeeb5-kube-api-access-b6jvc\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:32.381729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:32.381668    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/276f2e0b-545d-416c-b629-9b028aabeeb5-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:33.060220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:33.060126    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"
Nov 03 18:07:33.060220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:33.060154    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5" event={"ID":"276f2e0b-545d-416c-b629-9b028aabeeb5","Type":"ContainerDied","Data":"b53f66bdfbebd8bdb692db54cedcd3caeefc8b18cde7f26cdd7b356ec90ea1e9"}
Nov 03 18:07:33.060420 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:33.060223    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b53f66bdfbebd8bdb692db54cedcd3caeefc8b18cde7f26cdd7b356ec90ea1e9"
Nov 03 18:07:36.993404 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:07:36.993360    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:07:42.029788 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:42.029752    4493 scope.go:122] "RemoveContainer" containerID="04a66f8d0559e883f84a8d46e906599a488b618325aa11666262d5e8c251e79c"
Nov 03 18:07:42.040847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:42.040821    4493 scope.go:122] "RemoveContainer" containerID="644120ed8180276c82cbe2fc035d3acc4d9d96efbfb5e5f157c6ea7a54673d4c"
Nov 03 18:07:42.050410 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:42.050392    4493 scope.go:122] "RemoveContainer" containerID="624778107ce668e83dfaf820ec60da247deda286fca5198fc92f37f76872e937"
Nov 03 18:07:43.857603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:43.857574    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"]
Nov 03 18:07:43.858059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:43.857801    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy" containerID="cri-o://bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e" gracePeriod=30
Nov 03 18:07:44.086184 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:44.086154    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"]
Nov 03 18:07:44.086381 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:44.086347    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" containerID="cri-o://53999f6cb2dd3b1ffd55321e0a172f84302d2ba773b368a9cc951047f98d6590" gracePeriod=30
Nov 03 18:07:44.794856 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:44.794824    4493 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:07:44.795074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:44.794923    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:07:49.195204 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.195167    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 18:07:49.335202 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335172    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-data\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.335375 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335227    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-istio-token\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-istio-token\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.335454 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335428    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-data" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.335512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335498    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-kube-api-access-lz7nk\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-kube-api-access-lz7nk\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.335585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335563    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-envoy\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.335734 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335717    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eca63299-6b3b-4242-9a3d-022fea22c541-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca63299-6b3b-4242-9a3d-022fea22c541-istio-podinfo\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.336974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335853    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-credential-socket\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.336974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.335987    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca63299-6b3b-4242-9a3d-022fea22c541-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca63299-6b3b-4242-9a3d-022fea22c541-istiod-ca-cert\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.336974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.336036    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-certs\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.336974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.336246    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-credential-socket" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.336974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.336525    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca63299-6b3b-4242-9a3d-022fea22c541-istiod-ca-cert" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:07:49.336974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.336590    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-socket\") pod \"eca63299-6b3b-4242-9a3d-022fea22c541\" (UID: \"eca63299-6b3b-4242-9a3d-022fea22c541\") "
Nov 03 18:07:49.336974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.336835    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-certs" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.337760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.337094    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-socket" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.338673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338551    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eca63299-6b3b-4242-9a3d-022fea22c541-istio-podinfo" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:07:49.338673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338626    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-istio-token" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:49.338673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338626    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-envoy" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.338915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338693    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-kube-api-access-lz7nk" pod "eca63299-6b3b-4242-9a3d-022fea22c541" (UID: "eca63299-6b3b-4242-9a3d-022fea22c541"). InnerVolumeSpecName "kube-api-access-lz7nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:49.338915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338711    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.338915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338732    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca63299-6b3b-4242-9a3d-022fea22c541-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.338915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338747    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.338915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338762    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.338915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.338771    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.344466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.344442    4493 generic.go:356] "Generic (PLEG): container finished" podID="eca63299-6b3b-4242-9a3d-022fea22c541" containerID="bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e" exitCode=0
Nov 03 18:07:49.344569 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.344506    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" event={"ID":"eca63299-6b3b-4242-9a3d-022fea22c541","Type":"ContainerDied","Data":"bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e"}
Nov 03 18:07:49.344569 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.344556    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"
Nov 03 18:07:49.344569 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.344566    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7" event={"ID":"eca63299-6b3b-4242-9a3d-022fea22c541","Type":"ContainerDied","Data":"300c54f7c4ac29b6df05685707e6cfe0a972fd15f3d59fa87632cbbeda69322b"}
Nov 03 18:07:49.344743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.344589    4493 scope.go:122] "RemoveContainer" containerID="bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e"
Nov 03 18:07:49.346225 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.346203    4493 generic.go:356] "Generic (PLEG): container finished" podID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerID="53999f6cb2dd3b1ffd55321e0a172f84302d2ba773b368a9cc951047f98d6590" exitCode=0
Nov 03 18:07:49.346311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.346242    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" event={"ID":"0cd43a67-2eea-4b95-8388-5a06f65d4ca4","Type":"ContainerDied","Data":"53999f6cb2dd3b1ffd55321e0a172f84302d2ba773b368a9cc951047f98d6590"}
Nov 03 18:07:49.375422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.375394    4493 scope.go:122] "RemoveContainer" containerID="bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e"
Nov 03 18:07:49.376721 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:07:49.376690    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e\": container with ID starting with bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e not found: ID does not exist" containerID="bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e"
Nov 03 18:07:49.376859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.376731    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e"} err="failed to get container status \"bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e\": rpc error: code = NotFound desc = could not find container \"bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e\": container with ID starting with bf4caf03cc25c0a9c507fa1651ef650ffec35a0619f6fc7337582eabdc02cf0e not found: ID does not exist"
Nov 03 18:07:49.379196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.379160    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 18:07:49.439621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439591    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-envoy\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.439621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439625    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-certs\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.439900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439648    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-socket\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.439900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439675    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-token\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-token\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.439900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439745    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istiod-ca-cert\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.439900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439763    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-podinfo\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.440127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439916    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-data\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.440127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.439932    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-certs" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.440127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.440033    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-credential-socket\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.440127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.440065    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-socket" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.440127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.440119    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-kube-api-access-gwc78\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-kube-api-access-gwc78\") pod \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\" (UID: \"0cd43a67-2eea-4b95-8388-5a06f65d4ca4\") "
Nov 03 18:07:49.440384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.440207    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-data" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.440384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.440148    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istiod-ca-cert" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:07:49.440384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.440342    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-credential-socket" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.441101 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441072    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7nk\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-kube-api-access-lz7nk\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441106    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca63299-6b3b-4242-9a3d-022fea22c541-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441121    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca63299-6b3b-4242-9a3d-022fea22c541-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441137    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441149    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441164    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca63299-6b3b-4242-9a3d-022fea22c541-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441194    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441208    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.441606 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.441223    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.442095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.442072    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-envoy" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:07:49.442260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.442236    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-podinfo" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:07:49.442304 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.442254    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-token" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:49.442469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.442443    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-kube-api-access-gwc78" pod "0cd43a67-2eea-4b95-8388-5a06f65d4ca4" (UID: "0cd43a67-2eea-4b95-8388-5a06f65d4ca4"). InnerVolumeSpecName "kube-api-access-gwc78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:07:49.494301 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.494269    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"]
Nov 03 18:07:49.500722 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.500690    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-zbrv7"]
Nov 03 18:07:49.542402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.542370    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.542402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.542397    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwc78\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-kube-api-access-gwc78\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.542402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.542410    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:49.542623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:49.542418    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd43a67-2eea-4b95-8388-5a06f65d4ca4-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:07:50.354312 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:50.354279    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"
Nov 03 18:07:50.354312 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:50.354295    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s" event={"ID":"0cd43a67-2eea-4b95-8388-5a06f65d4ca4","Type":"ContainerDied","Data":"e80ed6587e298befdb21e858d92a1831219646e372db64b56cb61cbebaac0f0f"}
Nov 03 18:07:50.354766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:50.354359    4493 scope.go:122] "RemoveContainer" containerID="53999f6cb2dd3b1ffd55321e0a172f84302d2ba773b368a9cc951047f98d6590"
Nov 03 18:07:50.580008 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:50.579973    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"]
Nov 03 18:07:50.630577 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:50.630498    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-lxb2s"]
Nov 03 18:07:50.714305 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:50.714271    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" path="/var/lib/kubelet/pods/0cd43a67-2eea-4b95-8388-5a06f65d4ca4/volumes"
Nov 03 18:07:50.714852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:07:50.714830    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" path="/var/lib/kubelet/pods/eca63299-6b3b-4242-9a3d-022fea22c541/volumes"
Nov 03 18:08:06.992527 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:08:06.992486    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:08:11.697771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.697733    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"]
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698253    4493 cpu_manager.go:414] "containerMap: removing container" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698265    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698284    4493 cpu_manager.go:414] "containerMap: removing container" podUID="276f2e0b-545d-416c-b629-9b028aabeeb5" containerName="revoke-keys"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698289    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276f2e0b-545d-416c-b629-9b028aabeeb5" containerName="revoke-keys"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698299    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698305    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698405    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="276f2e0b-545d-416c-b629-9b028aabeeb5" containerName="revoke-keys"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698414    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd43a67-2eea-4b95-8388-5a06f65d4ca4" containerName="istio-proxy"
Nov 03 18:08:11.700413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.698427    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca63299-6b3b-4242-9a3d-022fea22c541" containerName="istio-proxy"
Nov 03 18:08:11.701966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.701943    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.770932 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.764210    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-640dc9-openshift-default-dockercfg-8lxnx\""
Nov 03 18:08:11.820901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.820490    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"]
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.876914    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjddd\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-kube-api-access-rjddd\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.876970    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cd61f-0b32-4936-9c60-158663ebe336-istiod-ca-cert\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.877008    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-certs\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.877134    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-credential-socket\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.877274    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-data\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.877341    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-envoy\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.877401    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-socket\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.877455    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-podinfo\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:11.877922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.877584    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-token\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993021    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-token\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993136    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjddd\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-kube-api-access-rjddd\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993170    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cd61f-0b32-4936-9c60-158663ebe336-istiod-ca-cert\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993211    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-certs\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993300    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-credential-socket\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993399    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-data\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993456    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-envoy\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993497    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-socket\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:11.993540    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-podinfo\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.012923    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-certs\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.014105    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cd61f-0b32-4936-9c60-158663ebe336-istiod-ca-cert\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.014343    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-data\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.014544    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-credential-socket\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.017458    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-envoy\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.022893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.017703    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-socket\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.032671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.032640    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-podinfo\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.078121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.078090    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-token\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.078488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.078462    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjddd\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-kube-api-access-rjddd\") pod \"e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") " pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.316744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.316647    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:12.757586 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:08:12.757554    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6cd61f_0b32_4936_9c60_158663ebe336.slice/crio-c3f496d8a00fa0b706e41399bb4950ac2f6d4c83fa012f44a88bbd3538a015a8 WatchSource:0}: Error finding container c3f496d8a00fa0b706e41399bb4950ac2f6d4c83fa012f44a88bbd3538a015a8: Status 404 returned error can't find the container with id c3f496d8a00fa0b706e41399bb4950ac2f6d4c83fa012f44a88bbd3538a015a8
Nov 03 18:08:12.758841 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.758816    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"]
Nov 03 18:08:12.760625 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.760586    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:08:12.760703 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.760674    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:08:12.760747 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:12.760715    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:08:13.515530 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:13.515485    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" event={"ID":"2b6cd61f-0b32-4936-9c60-158663ebe336","Type":"ContainerStarted","Data":"067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658"}
Nov 03 18:08:13.515739 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:13.515548    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" event={"ID":"2b6cd61f-0b32-4936-9c60-158663ebe336","Type":"ContainerStarted","Data":"c3f496d8a00fa0b706e41399bb4950ac2f6d4c83fa012f44a88bbd3538a015a8"}
Nov 03 18:08:13.694628 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:13.694576    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" podStartSLOduration=2.694561517 podStartE2EDuration="2.694561517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:13.693915214 +0000 UTC m=+2073.153086222" watchObservedRunningTime="2026-11-03 18:08:13.694561517 +0000 UTC m=+2073.153732503"
Nov 03 18:08:14.317622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:14.317594    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:14.324195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:14.324171    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:14.528471 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:14.528438    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:14.529522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:14.529500    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:08:21.228995 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.228960    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"]
Nov 03 18:08:21.276509 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.276478    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:21.285264 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.285229    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"]
Nov 03 18:08:21.340468 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.340431    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-640dc9\""
Nov 03 18:08:21.355376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.355344    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-640dc9-dockercfg-dwrjm\""
Nov 03 18:08:21.458091 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.458056    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nfw\" (UniqueName: \"kubernetes.io/projected/13316d7a-0f35-41af-ae5c-52febd2e3132-kube-api-access-w9nfw\") pod \"payload-processing-e2e-api-b-640dc9-d4f98757-ppznm\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") " pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:21.458294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.458118    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13316d7a-0f35-41af-ae5c-52febd2e3132-plugins-config-volume\") pod \"payload-processing-e2e-api-b-640dc9-d4f98757-ppznm\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") " pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:21.559305 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.559234    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nfw\" (UniqueName: \"kubernetes.io/projected/13316d7a-0f35-41af-ae5c-52febd2e3132-kube-api-access-w9nfw\") pod \"payload-processing-e2e-api-b-640dc9-d4f98757-ppznm\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") " pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:21.559305 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.559304    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13316d7a-0f35-41af-ae5c-52febd2e3132-plugins-config-volume\") pod \"payload-processing-e2e-api-b-640dc9-d4f98757-ppznm\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") " pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:21.559991 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.559973    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13316d7a-0f35-41af-ae5c-52febd2e3132-plugins-config-volume\") pod \"payload-processing-e2e-api-b-640dc9-d4f98757-ppznm\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") " pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:21.625285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.625253    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nfw\" (UniqueName: \"kubernetes.io/projected/13316d7a-0f35-41af-ae5c-52febd2e3132-kube-api-access-w9nfw\") pod \"payload-processing-e2e-api-b-640dc9-d4f98757-ppznm\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") " pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:21.918748 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:21.918674    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:22.317156 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:22.317123    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"]
Nov 03 18:08:22.318367 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:08:22.318339    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13316d7a_0f35_41af_ae5c_52febd2e3132.slice/crio-a37cd187d6ccca168b7d0e09b665cc2aaf449ca2b4263f4757b9dc5b52cea618 WatchSource:0}: Error finding container a37cd187d6ccca168b7d0e09b665cc2aaf449ca2b4263f4757b9dc5b52cea618: Status 404 returned error can't find the container with id a37cd187d6ccca168b7d0e09b665cc2aaf449ca2b4263f4757b9dc5b52cea618
Nov 03 18:08:22.582112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:22.582022    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm" event={"ID":"13316d7a-0f35-41af-ae5c-52febd2e3132","Type":"ContainerStarted","Data":"633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9"}
Nov 03 18:08:22.582112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:22.582072    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm" event={"ID":"13316d7a-0f35-41af-ae5c-52febd2e3132","Type":"ContainerStarted","Data":"a37cd187d6ccca168b7d0e09b665cc2aaf449ca2b4263f4757b9dc5b52cea618"}
Nov 03 18:08:22.582112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:22.582093    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:22.703249 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:22.703202    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm" podStartSLOduration=1.70318846 podStartE2EDuration="1.70318846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:22.70280447 +0000 UTC m=+2082.161975467" watchObservedRunningTime="2026-11-03 18:08:22.70318846 +0000 UTC m=+2082.162359446"
Nov 03 18:08:33.592168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:33.592137    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:08:36.994121 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:08:36.994078    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:08:37.059118 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:37.059085    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"]
Nov 03 18:08:37.111045 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:37.111011    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c430873-5pdbn"]
Nov 03 18:08:38.782844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:38.782808    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eae40ab-5a22-414f-9792-82b31b272ecc" path="/var/lib/kubelet/pods/7eae40ab-5a22-414f-9792-82b31b272ecc/volumes"
Nov 03 18:08:40.474030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.473997    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"]
Nov 03 18:08:40.478686 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.478661    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:40.516493 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.516459    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453bcafe-e700-4328-aa65-f956dfad21ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:40.516672 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.516498    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x855x\" (UniqueName: \"kubernetes.io/projected/453bcafe-e700-4328-aa65-f956dfad21ce-kube-api-access-x855x\") pod \"payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:40.538726 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.538628    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"]
Nov 03 18:08:40.584534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.584496    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f5100c\""
Nov 03 18:08:40.619135 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.619074    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453bcafe-e700-4328-aa65-f956dfad21ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:40.622694 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.622652    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x855x\" (UniqueName: \"kubernetes.io/projected/453bcafe-e700-4328-aa65-f956dfad21ce-kube-api-access-x855x\") pod \"payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:40.649848 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.649345    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453bcafe-e700-4328-aa65-f956dfad21ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:40.741671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.741223    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x855x\" (UniqueName: \"kubernetes.io/projected/453bcafe-e700-4328-aa65-f956dfad21ce-kube-api-access-x855x\") pod \"payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:40.794330 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:40.794297    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:41.181559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:41.181526    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"]
Nov 03 18:08:41.183017 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:08:41.182990    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453bcafe_e700_4328_aa65_f956dfad21ce.slice/crio-1d91dd3482ece47d3e83d35a29bfd881366a2af914a76be1c28ea8eb25ab14d0 WatchSource:0}: Error finding container 1d91dd3482ece47d3e83d35a29bfd881366a2af914a76be1c28ea8eb25ab14d0: Status 404 returned error can't find the container with id 1d91dd3482ece47d3e83d35a29bfd881366a2af914a76be1c28ea8eb25ab14d0
Nov 03 18:08:41.806216 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:41.806179    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9" event={"ID":"453bcafe-e700-4328-aa65-f956dfad21ce","Type":"ContainerStarted","Data":"57b8e8eb6a73040d038dbd9bd63be4a6d194be794e6a1f4b9c5b8501f00872c6"}
Nov 03 18:08:41.806675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:41.806250    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9" event={"ID":"453bcafe-e700-4328-aa65-f956dfad21ce","Type":"ContainerStarted","Data":"1d91dd3482ece47d3e83d35a29bfd881366a2af914a76be1c28ea8eb25ab14d0"}
Nov 03 18:08:41.806675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:41.806402    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:08:41.927127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:41.927070    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9" podStartSLOduration=1.9270565039999998 podStartE2EDuration="1.927056504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:41.925630001 +0000 UTC m=+2101.384800989" watchObservedRunningTime="2026-11-03 18:08:41.927056504 +0000 UTC m=+2101.386227490"
Nov 03 18:08:42.088224 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:42.088116    4493 scope.go:122] "RemoveContainer" containerID="35c8e9b146b260abb029ede2f5f3f90f13c24c467b4d43607aeb62c1a02b0e62"
Nov 03 18:08:43.107102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.107057    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"]
Nov 03 18:08:43.112662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.112639    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.171269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.171233    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"]
Nov 03 18:08:43.190915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.190860    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smr82\" (UniqueName: \"kubernetes.io/projected/feee9e96-148a-4a0f-b5e3-ffa480675f04-kube-api-access-smr82\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.191135 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.191112    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.191226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.191157    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.216602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.216543    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:08:43.216602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.216570    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:08:43.216602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.216569    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:08:43.216821 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.216708    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:08:43.292966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.292933    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.293148 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.292989    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.293195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.293143    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smr82\" (UniqueName: \"kubernetes.io/projected/feee9e96-148a-4a0f-b5e3-ffa480675f04-kube-api-access-smr82\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.293657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.293638    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.295613 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.295597    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.356957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.356913    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smr82\" (UniqueName: \"kubernetes.io/projected/feee9e96-148a-4a0f-b5e3-ffa480675f04-kube-api-access-smr82\") pod \"maas-api-revoke-sub-697aa93a-c7ldb\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.428347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.428271    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:43.814927 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:43.814892    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"]
Nov 03 18:08:43.815567 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:08:43.815539    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeee9e96_148a_4a0f_b5e3_ffa480675f04.slice/crio-dba011a71ed2a436aa00a31bcbf8ae5ecba8179fddf556012f8c250b784e13a0 WatchSource:0}: Error finding container dba011a71ed2a436aa00a31bcbf8ae5ecba8179fddf556012f8c250b784e13a0: Status 404 returned error can't find the container with id dba011a71ed2a436aa00a31bcbf8ae5ecba8179fddf556012f8c250b784e13a0
Nov 03 18:08:44.827834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:44.827799    4493 generic.go:356] "Generic (PLEG): container finished" podID="feee9e96-148a-4a0f-b5e3-ffa480675f04" containerID="505a237f5caa4f606ad123ee0998fdce926310228d9084776460d63bd232a4ec" exitCode=0
Nov 03 18:08:44.828280 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:44.827904    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb" event={"ID":"feee9e96-148a-4a0f-b5e3-ffa480675f04","Type":"ContainerDied","Data":"505a237f5caa4f606ad123ee0998fdce926310228d9084776460d63bd232a4ec"}
Nov 03 18:08:44.828280 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:44.827978    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb" event={"ID":"feee9e96-148a-4a0f-b5e3-ffa480675f04","Type":"ContainerStarted","Data":"dba011a71ed2a436aa00a31bcbf8ae5ecba8179fddf556012f8c250b784e13a0"}
Nov 03 18:08:46.003504 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.003474    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:46.135919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.135814    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-openshift-ca\") pod \"feee9e96-148a-4a0f-b5e3-ffa480675f04\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") "
Nov 03 18:08:46.136077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.135973    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-xks-ca\") pod \"feee9e96-148a-4a0f-b5e3-ffa480675f04\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") "
Nov 03 18:08:46.136154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.136097    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feee9e96-148a-4a0f-b5e3-ffa480675f04-kube-api-access-smr82\" (UniqueName: \"kubernetes.io/projected/feee9e96-148a-4a0f-b5e3-ffa480675f04-kube-api-access-smr82\") pod \"feee9e96-148a-4a0f-b5e3-ffa480675f04\" (UID: \"feee9e96-148a-4a0f-b5e3-ffa480675f04\") "
Nov 03 18:08:46.136215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.136201    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-openshift-ca" pod "feee9e96-148a-4a0f-b5e3-ffa480675f04" (UID: "feee9e96-148a-4a0f-b5e3-ffa480675f04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:08:46.136673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.136645    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:08:46.138744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.138714    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feee9e96-148a-4a0f-b5e3-ffa480675f04-kube-api-access-smr82" pod "feee9e96-148a-4a0f-b5e3-ffa480675f04" (UID: "feee9e96-148a-4a0f-b5e3-ffa480675f04"). InnerVolumeSpecName "kube-api-access-smr82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:08:46.138893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.138797    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-xks-ca" pod "feee9e96-148a-4a0f-b5e3-ffa480675f04" (UID: "feee9e96-148a-4a0f-b5e3-ffa480675f04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:08:46.238433 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.238399    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smr82\" (UniqueName: \"kubernetes.io/projected/feee9e96-148a-4a0f-b5e3-ffa480675f04-kube-api-access-smr82\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:08:46.238433 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.238428    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feee9e96-148a-4a0f-b5e3-ffa480675f04-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:08:46.841278 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.841244    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"
Nov 03 18:08:46.841433 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.841280    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb" event={"ID":"feee9e96-148a-4a0f-b5e3-ffa480675f04","Type":"ContainerDied","Data":"dba011a71ed2a436aa00a31bcbf8ae5ecba8179fddf556012f8c250b784e13a0"}
Nov 03 18:08:46.841433 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:46.841331    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dba011a71ed2a436aa00a31bcbf8ae5ecba8179fddf556012f8c250b784e13a0"
Nov 03 18:08:52.815488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:08:52.815410    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:09:06.992677 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:09:06.992639    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:09:30.887540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:30.887504    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"]
Nov 03 18:09:30.888078 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:30.887730    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" containerID="cri-o://6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246" gracePeriod=30
Nov 03 18:09:36.172623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.172591    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 18:09:36.192366 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.192329    4493 generic.go:356] "Generic (PLEG): container finished" podID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerID="6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246" exitCode=0
Nov 03 18:09:36.192503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.192378    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"
Nov 03 18:09:36.192503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.192388    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" event={"ID":"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7","Type":"ContainerDied","Data":"6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246"}
Nov 03 18:09:36.192503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.192471    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" event={"ID":"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7","Type":"ContainerDied","Data":"7baf1e754c174196c896f88ae37de5c66d7ffadd78b473ac64ad249bda1cf84a"}
Nov 03 18:09:36.192618 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.192503    4493 scope.go:122] "RemoveContainer" containerID="6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246"
Nov 03 18:09:36.204625 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.204603    4493 scope.go:122] "RemoveContainer" containerID="6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246"
Nov 03 18:09:36.205019 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:09:36.204985    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246\": container with ID starting with 6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246 not found: ID does not exist" containerID="6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246"
Nov 03 18:09:36.205125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.205030    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246"} err="failed to get container status \"6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246\": rpc error: code = NotFound desc = could not find container \"6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246\": container with ID starting with 6d6c6fc8daf8c0954f1cf720d38255c7390466e5c0996ac18db6715fdbbe5246 not found: ID does not exist"
Nov 03 18:09:36.254007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.253978    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-credential-socket\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254087    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istiod-ca-cert\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254142    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-certs\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254169    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-data\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254281 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254203    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-envoy\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254281 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254225    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-token\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-token\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254281 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254232    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-credential-socket" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:36.254430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254394    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-kube-api-access-bpxdg\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254486    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-certs" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:36.254516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254506    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-podinfo\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254511    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-data" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:36.254621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254547    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-socket\") pod \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\" (UID: \"c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7\") "
Nov 03 18:09:36.254621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254565    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istiod-ca-cert" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:36.254854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.254829    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-socket" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:36.255227 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.255197    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.255227 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.255223    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.255401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.255233    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.255401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.255242    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.255401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.255250    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.256798 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.256772    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-envoy" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:36.256911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.256774    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-podinfo" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:36.257103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.257084    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-token" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:36.257167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.257129    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-kube-api-access-bpxdg" pod "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" (UID: "c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7"). InnerVolumeSpecName "kube-api-access-bpxdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:36.356720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.356686    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-kube-api-access-bpxdg\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.356720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.356715    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.356720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.356725    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.356968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.356733    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:36.620573 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.620534    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"]
Nov 03 18:09:36.670422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.670387    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68"]
Nov 03 18:09:36.714083 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:36.714048    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" path="/var/lib/kubelet/pods/c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7/volumes"
Nov 03 18:09:36.992679 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:09:36.992637    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:09:37.101847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:37.101812    4493 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:09:37.102061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:37.101893    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8cz68" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:09:39.246261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.246229    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"]
Nov 03 18:09:39.246775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.246760    4493 cpu_manager.go:414] "containerMap: removing container" podUID="feee9e96-148a-4a0f-b5e3-ffa480675f04" containerName="revoke-keys"
Nov 03 18:09:39.246823 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.246776    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feee9e96-148a-4a0f-b5e3-ffa480675f04" containerName="revoke-keys"
Nov 03 18:09:39.246823 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.246801    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy"
Nov 03 18:09:39.246823 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.246806    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy"
Nov 03 18:09:39.246955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.246919    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74829e7-56b7-4fb4-8ee2-f55eef2bb8c7" containerName="istio-proxy"
Nov 03 18:09:39.246955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.246936    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="feee9e96-148a-4a0f-b5e3-ffa480675f04" containerName="revoke-keys"
Nov 03 18:09:39.251243 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.251222    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.309437 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.309411    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"]
Nov 03 18:09:39.356540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.356506    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:09:39.356540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.356521    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:09:39.356540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.356540    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:09:39.356540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.356507    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:09:39.397168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.397135    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.397328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.397181    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2tkz\" (UniqueName: \"kubernetes.io/projected/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-kube-api-access-k2tkz\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.397328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.397271    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.498515 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.498432    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.498515 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.498487    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tkz\" (UniqueName: \"kubernetes.io/projected/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-kube-api-access-k2tkz\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.498714 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.498539    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.499231 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.499211    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.501197 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.501181    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.562720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.562686    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tkz\" (UniqueName: \"kubernetes.io/projected/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-kube-api-access-k2tkz\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.564559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.564539    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:39.952627 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:39.952593    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"]
Nov 03 18:09:40.219669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:40.219633    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts" event={"ID":"d5eb4ec0-59d5-47a7-a39d-f6fab041c412","Type":"ContainerStarted","Data":"d880c33bbc59666f2cabbbfb0bbed6a9736cd93f96d947b588120600a7c261db"}
Nov 03 18:09:41.228122 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:41.228084    4493 generic.go:356] "Generic (PLEG): container finished" podID="d5eb4ec0-59d5-47a7-a39d-f6fab041c412" containerID="fffec639da95e83bf665e49a5bc9f65b66f26c3d461890c1f6c200e4a15fc71e" exitCode=0
Nov 03 18:09:41.228535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:41.228167    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts" event={"ID":"d5eb4ec0-59d5-47a7-a39d-f6fab041c412","Type":"ContainerDied","Data":"fffec639da95e83bf665e49a5bc9f65b66f26c3d461890c1f6c200e4a15fc71e"}
Nov 03 18:09:42.152932 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.152903    4493 scope.go:122] "RemoveContainer" containerID="f58c57755722cfa8b701278ccc462edc47b046594fbc9bcbce67bc93a7f31bcc"
Nov 03 18:09:42.162945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.162863    4493 scope.go:122] "RemoveContainer" containerID="c519d74a3bb3bd4fdb8e85c3b2165cf68aed9d92f4d3cbe519bab0ed9f23507e"
Nov 03 18:09:42.171791 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.171767    4493 scope.go:122] "RemoveContainer" containerID="92e06e25bea80c98317489e6a8ad8ae3ba0141f08583fb0fa0c4bd0e2040ede6"
Nov 03 18:09:42.402798 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.402775    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:42.541951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.541922    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-kube-api-access-k2tkz\" (UniqueName: \"kubernetes.io/projected/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-kube-api-access-k2tkz\") pod \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") "
Nov 03 18:09:42.542406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.542023    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-xks-ca\") pod \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") "
Nov 03 18:09:42.542406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.542078    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-openshift-ca\") pod \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\" (UID: \"d5eb4ec0-59d5-47a7-a39d-f6fab041c412\") "
Nov 03 18:09:42.542600 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.542500    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-openshift-ca" pod "d5eb4ec0-59d5-47a7-a39d-f6fab041c412" (UID: "d5eb4ec0-59d5-47a7-a39d-f6fab041c412"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:42.543025 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.543002    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:42.544387 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.544363    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-kube-api-access-k2tkz" pod "d5eb4ec0-59d5-47a7-a39d-f6fab041c412" (UID: "d5eb4ec0-59d5-47a7-a39d-f6fab041c412"). InnerVolumeSpecName "kube-api-access-k2tkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:42.544828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.544813    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-xks-ca" pod "d5eb4ec0-59d5-47a7-a39d-f6fab041c412" (UID: "d5eb4ec0-59d5-47a7-a39d-f6fab041c412"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:42.644067 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.644026    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:42.644067 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:42.644066    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2tkz\" (UniqueName: \"kubernetes.io/projected/d5eb4ec0-59d5-47a7-a39d-f6fab041c412-kube-api-access-k2tkz\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:43.250584 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:43.250546    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts" event={"ID":"d5eb4ec0-59d5-47a7-a39d-f6fab041c412","Type":"ContainerDied","Data":"d880c33bbc59666f2cabbbfb0bbed6a9736cd93f96d947b588120600a7c261db"}
Nov 03 18:09:43.250584 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:43.250585    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"
Nov 03 18:09:43.250805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:43.250630    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d880c33bbc59666f2cabbbfb0bbed6a9736cd93f96d947b588120600a7c261db"
Nov 03 18:09:44.274128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.274095    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"]
Nov 03 18:09:44.274833 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.274804    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d5eb4ec0-59d5-47a7-a39d-f6fab041c412" containerName="revoke-keys"
Nov 03 18:09:44.274833 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.274822    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5eb4ec0-59d5-47a7-a39d-f6fab041c412" containerName="revoke-keys"
Nov 03 18:09:44.275079 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.275062    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5eb4ec0-59d5-47a7-a39d-f6fab041c412" containerName="revoke-keys"
Nov 03 18:09:44.280065 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.280042    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.336753 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.336721    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"]
Nov 03 18:09:44.383161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.383131    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:09:44.383161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.383137    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:09:44.383358 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.383197    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:09:44.384770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.384748    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:09:44.466735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.466700    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.466735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.466738    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.467009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.466846    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntz7\" (UniqueName: \"kubernetes.io/projected/bd52f69f-ac8e-4143-94e5-4777c9728571-kube-api-access-vntz7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.568671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.568595    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.568671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.568632    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.568901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.568688    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntz7\" (UniqueName: \"kubernetes.io/projected/bd52f69f-ac8e-4143-94e5-4777c9728571-kube-api-access-vntz7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.569402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.569376    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.571190 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.571168    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.632764 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.632727    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntz7\" (UniqueName: \"kubernetes.io/projected/bd52f69f-ac8e-4143-94e5-4777c9728571-kube-api-access-vntz7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:44.897003 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:44.896931    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:45.284067 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:45.284038    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"]
Nov 03 18:09:45.284839 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:09:45.284815    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd52f69f_ac8e_4143_94e5_4777c9728571.slice/crio-4f9fecf61800c102c13b03c12cf989c511ecc6427e3ed1e06d77afde2d2cd8ae WatchSource:0}: Error finding container 4f9fecf61800c102c13b03c12cf989c511ecc6427e3ed1e06d77afde2d2cd8ae: Status 404 returned error can't find the container with id 4f9fecf61800c102c13b03c12cf989c511ecc6427e3ed1e06d77afde2d2cd8ae
Nov 03 18:09:45.401679 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:45.401639    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"]
Nov 03 18:09:45.401918 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:45.401863    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" containerID="cri-o://139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470" gracePeriod=30
Nov 03 18:09:45.452042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:45.451997    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"]
Nov 03 18:09:45.452244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:45.452206    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" containerID="cri-o://219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51" gracePeriod=30
Nov 03 18:09:46.281960 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:46.281916    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv" event={"ID":"bd52f69f-ac8e-4143-94e5-4777c9728571","Type":"ContainerStarted","Data":"0060d75b40c25c4a4e84a8e974752565ee2595dbd74d5b25e8ce529e97b22fd0"}
Nov 03 18:09:46.282146 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:46.282019    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv" event={"ID":"bd52f69f-ac8e-4143-94e5-4777c9728571","Type":"ContainerStarted","Data":"4f9fecf61800c102c13b03c12cf989c511ecc6427e3ed1e06d77afde2d2cd8ae"}
Nov 03 18:09:46.378784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:46.378755    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:09:46.379145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:46.378810    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:09:46.402361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:46.402316    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv" podStartSLOduration=2.402304891 podStartE2EDuration="2.402304891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:46.401756705 +0000 UTC m=+2165.860927692" watchObservedRunningTime="2026-11-03 18:09:46.402304891 +0000 UTC m=+2165.861475877"
Nov 03 18:09:47.166219 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.166186    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"]
Nov 03 18:09:47.166463 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.166408    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" podUID="9b85984f-07ac-4d69-bbce-bd760c705376" containerName="discovery" containerID="cri-o://fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9" gracePeriod=30
Nov 03 18:09:47.295153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.295105    4493 generic.go:356] "Generic (PLEG): container finished" podID="bd52f69f-ac8e-4143-94e5-4777c9728571" containerID="0060d75b40c25c4a4e84a8e974752565ee2595dbd74d5b25e8ce529e97b22fd0" exitCode=0
Nov 03 18:09:47.295324 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.295201    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv" event={"ID":"bd52f69f-ac8e-4143-94e5-4777c9728571","Type":"ContainerDied","Data":"0060d75b40c25c4a4e84a8e974752565ee2595dbd74d5b25e8ce529e97b22fd0"}
Nov 03 18:09:47.466604 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.466577    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:09:47.576194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.576154    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-kubeconfig\") pod \"9b85984f-07ac-4d69-bbce-bd760c705376\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") "
Nov 03 18:09:47.576968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.576944    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b85984f-07ac-4d69-bbce-bd760c705376-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b85984f-07ac-4d69-bbce-bd760c705376-local-certs\") pod \"9b85984f-07ac-4d69-bbce-bd760c705376\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") "
Nov 03 18:09:47.577501 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.577483    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-dns-cert\") pod \"9b85984f-07ac-4d69-bbce-bd760c705376\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") "
Nov 03 18:09:47.577972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.577954    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-kube-api-access-mg7d5\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-kube-api-access-mg7d5\") pod \"9b85984f-07ac-4d69-bbce-bd760c705376\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") "
Nov 03 18:09:47.578428 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.578410    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-istio-token\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-istio-token\") pod \"9b85984f-07ac-4d69-bbce-bd760c705376\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") "
Nov 03 18:09:47.578998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.578978    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-ca-configmap\") pod \"9b85984f-07ac-4d69-bbce-bd760c705376\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") "
Nov 03 18:09:47.579804 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.579474    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-ca-configmap" pod "9b85984f-07ac-4d69-bbce-bd760c705376" (UID: "9b85984f-07ac-4d69-bbce-bd760c705376"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:47.584570 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.584552    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-cacerts\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-cacerts\") pod \"9b85984f-07ac-4d69-bbce-bd760c705376\" (UID: \"9b85984f-07ac-4d69-bbce-bd760c705376\") "
Nov 03 18:09:47.585814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.585761    4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-ca-configmap\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:47.588103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.588064    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-kubeconfig" pod "9b85984f-07ac-4d69-bbce-bd760c705376" (UID: "9b85984f-07ac-4d69-bbce-bd760c705376"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:47.591273 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.591237    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b85984f-07ac-4d69-bbce-bd760c705376-local-certs" pod "9b85984f-07ac-4d69-bbce-bd760c705376" (UID: "9b85984f-07ac-4d69-bbce-bd760c705376"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:47.601272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.601225    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-istio-token" pod "9b85984f-07ac-4d69-bbce-bd760c705376" (UID: "9b85984f-07ac-4d69-bbce-bd760c705376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:47.603643 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.603597    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-cacerts" pod "9b85984f-07ac-4d69-bbce-bd760c705376" (UID: "9b85984f-07ac-4d69-bbce-bd760c705376"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:47.614184 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.613078    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-kube-api-access-mg7d5" pod "9b85984f-07ac-4d69-bbce-bd760c705376" (UID: "9b85984f-07ac-4d69-bbce-bd760c705376"). InnerVolumeSpecName "kube-api-access-mg7d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:47.614184 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.613169    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-dns-cert" pod "9b85984f-07ac-4d69-bbce-bd760c705376" (UID: "9b85984f-07ac-4d69-bbce-bd760c705376"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:47.651068 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.650999    4493 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:09:47.651240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.651108    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:09:47.687852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.687786    4493 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-cacerts\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:47.687852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.687813    4493 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-kubeconfig\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:47.687852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.687824    4493 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b85984f-07ac-4d69-bbce-bd760c705376-local-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:47.687852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.687837    4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b85984f-07ac-4d69-bbce-bd760c705376-istio-csr-dns-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:47.687852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.687850    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7d5\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-kube-api-access-mg7d5\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:47.688125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:47.687862    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b85984f-07ac-4d69-bbce-bd760c705376-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:48.303098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.303063    4493 generic.go:356] "Generic (PLEG): container finished" podID="9b85984f-07ac-4d69-bbce-bd760c705376" containerID="fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9" exitCode=0
Nov 03 18:09:48.303277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.303115    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"
Nov 03 18:09:48.303277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.303143    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" event={"ID":"9b85984f-07ac-4d69-bbce-bd760c705376","Type":"ContainerDied","Data":"fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9"}
Nov 03 18:09:48.303277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.303198    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" event={"ID":"9b85984f-07ac-4d69-bbce-bd760c705376","Type":"ContainerDied","Data":"a9dff895231a0f4b6c0e0c6bf39a9bf7aecb0cf2034d35071c71cf8b2af80d4b"}
Nov 03 18:09:48.303277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.303218    4493 scope.go:122] "RemoveContainer" containerID="fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9"
Nov 03 18:09:48.313982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.313958    4493 scope.go:122] "RemoveContainer" containerID="fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9"
Nov 03 18:09:48.314308 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:09:48.314278    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9\": container with ID starting with fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9 not found: ID does not exist" containerID="fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9"
Nov 03 18:09:48.314398 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.314316    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9"} err="failed to get container status \"fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9\": rpc error: code = NotFound desc = could not find container \"fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9\": container with ID starting with fd02b488df681d8881823ba360e23f92a44f9d1394097101a9639c329098abd9 not found: ID does not exist"
Nov 03 18:09:48.462914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.462888    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:48.527704 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.527673    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"]
Nov 03 18:09:48.577996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.577917    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq"]
Nov 03 18:09:48.600403 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.600376    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-openshift-ca\") pod \"bd52f69f-ac8e-4143-94e5-4777c9728571\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") "
Nov 03 18:09:48.600558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.600448    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-xks-ca\") pod \"bd52f69f-ac8e-4143-94e5-4777c9728571\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") "
Nov 03 18:09:48.600558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.600543    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd52f69f-ac8e-4143-94e5-4777c9728571-kube-api-access-vntz7\" (UniqueName: \"kubernetes.io/projected/bd52f69f-ac8e-4143-94e5-4777c9728571-kube-api-access-vntz7\") pod \"bd52f69f-ac8e-4143-94e5-4777c9728571\" (UID: \"bd52f69f-ac8e-4143-94e5-4777c9728571\") "
Nov 03 18:09:48.600788 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.600767    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-openshift-ca" pod "bd52f69f-ac8e-4143-94e5-4777c9728571" (UID: "bd52f69f-ac8e-4143-94e5-4777c9728571"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:48.601144 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.601122    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:48.602765 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.602743    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd52f69f-ac8e-4143-94e5-4777c9728571-kube-api-access-vntz7" pod "bd52f69f-ac8e-4143-94e5-4777c9728571" (UID: "bd52f69f-ac8e-4143-94e5-4777c9728571"). InnerVolumeSpecName "kube-api-access-vntz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:48.602841 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.602783    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-xks-ca" pod "bd52f69f-ac8e-4143-94e5-4777c9728571" (UID: "bd52f69f-ac8e-4143-94e5-4777c9728571"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:09:48.702817 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.702791    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd52f69f-ac8e-4143-94e5-4777c9728571-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:48.702817 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.702817    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vntz7\" (UniqueName: \"kubernetes.io/projected/bd52f69f-ac8e-4143-94e5-4777c9728571-kube-api-access-vntz7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:48.713971 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:48.713944    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b85984f-07ac-4d69-bbce-bd760c705376" path="/var/lib/kubelet/pods/9b85984f-07ac-4d69-bbce-bd760c705376/volumes"
Nov 03 18:09:49.311832 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:49.311796    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv" event={"ID":"bd52f69f-ac8e-4143-94e5-4777c9728571","Type":"ContainerDied","Data":"4f9fecf61800c102c13b03c12cf989c511ecc6427e3ed1e06d77afde2d2cd8ae"}
Nov 03 18:09:49.312036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:49.311857    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f9fecf61800c102c13b03c12cf989c511ecc6427e3ed1e06d77afde2d2cd8ae"
Nov 03 18:09:49.312036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:49.311812    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"
Nov 03 18:09:50.781334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.781302    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 18:09:50.785488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.785456    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 18:09:50.830844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.830810    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-certs\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831020 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.830865    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istiod-ca-cert\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831020 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.830956    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-podinfo\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831066    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-data\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831129    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-certs" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.831285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831167    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-envoy\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831239    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-socket\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831395 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831300    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-kube-api-access-kp6qq\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-kube-api-access-kp6qq\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831395 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831322    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-data" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.831395 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831357    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-credential-socket\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831557 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831406    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-token\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-token\") pod \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\" (UID: \"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819\") "
Nov 03 18:09:50.831834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831354    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istiod-ca-cert" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:50.831834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.831784    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-socket" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.832595 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.832110    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.832595 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.832136    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.832595 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.832149    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.832595 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.832161    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.834957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.834930    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-credential-socket" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.835908 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.835762    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-envoy" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.835908 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.835855    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-podinfo" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:50.836076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.836054    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-kube-api-access-kp6qq" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "kube-api-access-kp6qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.836193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.836168    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-token" pod "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" (UID: "d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.932764 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.932678    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-credential-socket\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.932764 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.932753    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-envoy\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.932797    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-data\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.932827    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-token\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.932999    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ce03c0-8b18-424e-892d-ebde780fb409-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce03c0-8b18-424e-892d-ebde780fb409-istiod-ca-cert\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933029    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-podinfo\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933028    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-credential-socket" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.933423 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933074    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-socket\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933423 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933151    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-certs\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933423 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933298    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ce03c0-8b18-424e-892d-ebde780fb409-istiod-ca-cert" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:09:50.933423 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933322    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-kube-api-access-jtxmp\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-kube-api-access-jtxmp\") pod \"a4ce03c0-8b18-424e-892d-ebde780fb409\" (UID: \"a4ce03c0-8b18-424e-892d-ebde780fb409\") "
Nov 03 18:09:50.933688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933525    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-data" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.933688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933610    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-certs" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.933688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.933655    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-socket" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.934046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934022    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934138 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934052    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934138 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934070    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934138 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934085    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce03c0-8b18-424e-892d-ebde780fb409-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934138 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934099    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934184    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934329 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934285    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934329 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934299    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp6qq\" (UniqueName: \"kubernetes.io/projected/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-kube-api-access-kp6qq\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934329 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934313    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.934329 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.934326    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:50.936282 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.936234    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-podinfo" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:09:50.936434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.936406    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-envoy" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:09:50.936568 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.936548    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-token" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:50.936981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:50.936952    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-kube-api-access-jtxmp" pod "a4ce03c0-8b18-424e-892d-ebde780fb409" (UID: "a4ce03c0-8b18-424e-892d-ebde780fb409"). InnerVolumeSpecName "kube-api-access-jtxmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:09:51.036066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.036029    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtxmp\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-kube-api-access-jtxmp\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.036066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.036060    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.036066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.036073    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.036306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.036085    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce03c0-8b18-424e-892d-ebde780fb409-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:09:51.323866 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.323829    4493 generic.go:356] "Generic (PLEG): container finished" podID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerID="139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470" exitCode=0
Nov 03 18:09:51.324060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.323889    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"
Nov 03 18:09:51.324060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.323902    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" event={"ID":"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819","Type":"ContainerDied","Data":"139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470"}
Nov 03 18:09:51.324060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.323966    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7" event={"ID":"d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819","Type":"ContainerDied","Data":"a17e42379afc1d9020289617b739375647fd8e30e3ddee139945944ecd5399ab"}
Nov 03 18:09:51.324060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.323989    4493 scope.go:122] "RemoveContainer" containerID="139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470"
Nov 03 18:09:51.325629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.325565    4493 generic.go:356] "Generic (PLEG): container finished" podID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerID="219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51" exitCode=0
Nov 03 18:09:51.325629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.325601    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" event={"ID":"a4ce03c0-8b18-424e-892d-ebde780fb409","Type":"ContainerDied","Data":"219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51"}
Nov 03 18:09:51.325629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.325614    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"
Nov 03 18:09:51.325629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.325632    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh" event={"ID":"a4ce03c0-8b18-424e-892d-ebde780fb409","Type":"ContainerDied","Data":"da52477d2d7ee1ac3a9223c3aa42ede188663b7fde3dd10db5cadaaf4293ca17"}
Nov 03 18:09:51.334375 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.334355    4493 scope.go:122] "RemoveContainer" containerID="139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470"
Nov 03 18:09:51.335796 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:09:51.335773    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470\": container with ID starting with 139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470 not found: ID does not exist" containerID="139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470"
Nov 03 18:09:51.335918 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.335803    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470"} err="failed to get container status \"139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470\": rpc error: code = NotFound desc = could not find container \"139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470\": container with ID starting with 139f1e22c5fc0fa5886ad201ba9f6450d0849cdded83815ea09891363173c470 not found: ID does not exist"
Nov 03 18:09:51.335918 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.335822    4493 scope.go:122] "RemoveContainer" containerID="219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51"
Nov 03 18:09:51.347033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.347014    4493 scope.go:122] "RemoveContainer" containerID="219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51"
Nov 03 18:09:51.347280 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:09:51.347259    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51\": container with ID starting with 219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51 not found: ID does not exist" containerID="219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51"
Nov 03 18:09:51.347340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.347292    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51"} err="failed to get container status \"219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51\": rpc error: code = NotFound desc = could not find container \"219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51\": container with ID starting with 219095733a38fd3de7a3cd92ada059fe7e51bf44ee34628debcc46114be63b51 not found: ID does not exist"
Nov 03 18:09:51.497146 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.497108    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"]
Nov 03 18:09:51.505885 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.505832    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-sknfh"]
Nov 03 18:09:51.617366 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.617286    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"]
Nov 03 18:09:51.623310 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:51.623282    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-xv8r7"]
Nov 03 18:09:52.360589 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:52.360549    4493 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5jzlq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:09:52.361058 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:52.360631    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5jzlq" podUID="9b85984f-07ac-4d69-bbce-bd760c705376" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.140:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:09:52.721007 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:52.720974    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" path="/var/lib/kubelet/pods/a4ce03c0-8b18-424e-892d-ebde780fb409/volumes"
Nov 03 18:09:52.721641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:09:52.721624    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" path="/var/lib/kubelet/pods/d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819/volumes"
Nov 03 18:10:06.992526 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:10:06.992489    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:10:11.059617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:11.059587    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"]
Nov 03 18:10:11.103159 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:11.103128    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812e3d9a-nkqmz"]
Nov 03 18:10:12.713171 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:12.713138    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceab99bb-e1bf-4339-aafc-139f70a76b69" path="/var/lib/kubelet/pods/ceab99bb-e1bf-4339-aafc-139f70a76b69/volumes"
Nov 03 18:10:20.269982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.269950    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"]
Nov 03 18:10:20.270506 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270490    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy"
Nov 03 18:10:20.270506 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270506    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy"
Nov 03 18:10:20.270629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270528    4493 cpu_manager.go:414] "containerMap: removing container" podUID="bd52f69f-ac8e-4143-94e5-4777c9728571" containerName="revoke-keys"
Nov 03 18:10:20.270629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270533    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd52f69f-ac8e-4143-94e5-4777c9728571" containerName="revoke-keys"
Nov 03 18:10:20.270629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270541    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy"
Nov 03 18:10:20.270629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270547    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy"
Nov 03 18:10:20.270629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270558    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9b85984f-07ac-4d69-bbce-bd760c705376" containerName="discovery"
Nov 03 18:10:20.270629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270563    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b85984f-07ac-4d69-bbce-bd760c705376" containerName="discovery"
Nov 03 18:10:20.270958 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270722    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d84132-5d5f-4e2e-92f6-6ac5cb8f8819" containerName="istio-proxy"
Nov 03 18:10:20.270958 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270742    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd52f69f-ac8e-4143-94e5-4777c9728571" containerName="revoke-keys"
Nov 03 18:10:20.270958 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270750    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ce03c0-8b18-424e-892d-ebde780fb409" containerName="istio-proxy"
Nov 03 18:10:20.270958 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.270762    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b85984f-07ac-4d69-bbce-bd760c705376" containerName="discovery"
Nov 03 18:10:20.275095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.275073    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.333178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.333146    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"]
Nov 03 18:10:20.353097 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.353067    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.353342 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.353314    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.353484 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.353456    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lch2t\" (UniqueName: \"kubernetes.io/projected/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-kube-api-access-lch2t\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.379695 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.379670    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:10:20.379842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.379753    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:10:20.424160 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.424087    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:10:20.430412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.430396    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:10:20.455136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.455104    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.455294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.455276    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.455355 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.455338    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lch2t\" (UniqueName: \"kubernetes.io/projected/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-kube-api-access-lch2t\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.455916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.455861    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.457635 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.457607    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.519255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.519227    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lch2t\" (UniqueName: \"kubernetes.io/projected/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-kube-api-access-lch2t\") pod \"maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.589204 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.589168    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:20.974102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:20.974071    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"]
Nov 03 18:10:20.977930 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:10:20.975601    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cb0d2e_89cf_496b_8eea_e1626ddc12cf.slice/crio-883a699a6388f8100e3eca992122e96aaacc37d01423dd14b424f8e9c756ff42 WatchSource:0}: Error finding container 883a699a6388f8100e3eca992122e96aaacc37d01423dd14b424f8e9c756ff42: Status 404 returned error can't find the container with id 883a699a6388f8100e3eca992122e96aaacc37d01423dd14b424f8e9c756ff42
Nov 03 18:10:21.515897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:21.515781    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx" event={"ID":"50cb0d2e-89cf-496b-8eea-e1626ddc12cf","Type":"ContainerStarted","Data":"a4dc2563656d14bb311b7a8ae8420b0e84a350e8eeda781edf3e1e5f52ecdf04"}
Nov 03 18:10:21.516318 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:21.515924    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx" event={"ID":"50cb0d2e-89cf-496b-8eea-e1626ddc12cf","Type":"ContainerStarted","Data":"883a699a6388f8100e3eca992122e96aaacc37d01423dd14b424f8e9c756ff42"}
Nov 03 18:10:21.640497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:21.640439    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx" podStartSLOduration=1.6404201440000001 podStartE2EDuration="1.640420144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:21.637091977 +0000 UTC m=+2201.096262963" watchObservedRunningTime="2026-11-03 18:10:21.640420144 +0000 UTC m=+2201.099591127"
Nov 03 18:10:22.525072 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:22.525040    4493 generic.go:356] "Generic (PLEG): container finished" podID="50cb0d2e-89cf-496b-8eea-e1626ddc12cf" containerID="a4dc2563656d14bb311b7a8ae8420b0e84a350e8eeda781edf3e1e5f52ecdf04" exitCode=0
Nov 03 18:10:22.525498 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:22.525128    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx" event={"ID":"50cb0d2e-89cf-496b-8eea-e1626ddc12cf","Type":"ContainerDied","Data":"a4dc2563656d14bb311b7a8ae8420b0e84a350e8eeda781edf3e1e5f52ecdf04"}
Nov 03 18:10:23.681606 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.681583    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:23.721108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.721078    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-kube-api-access-lch2t\" (UniqueName: \"kubernetes.io/projected/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-kube-api-access-lch2t\") pod \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") "
Nov 03 18:10:23.721246 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.721139    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-openshift-ca\") pod \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") "
Nov 03 18:10:23.721246 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.721195    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-xks-ca\") pod \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\" (UID: \"50cb0d2e-89cf-496b-8eea-e1626ddc12cf\") "
Nov 03 18:10:23.721537 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.721504    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-openshift-ca" pod "50cb0d2e-89cf-496b-8eea-e1626ddc12cf" (UID: "50cb0d2e-89cf-496b-8eea-e1626ddc12cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:10:23.722142 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.722118    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:23.723254 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.723228    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-xks-ca" pod "50cb0d2e-89cf-496b-8eea-e1626ddc12cf" (UID: "50cb0d2e-89cf-496b-8eea-e1626ddc12cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:10:23.723379 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.723354    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-kube-api-access-lch2t" pod "50cb0d2e-89cf-496b-8eea-e1626ddc12cf" (UID: "50cb0d2e-89cf-496b-8eea-e1626ddc12cf"). InnerVolumeSpecName "kube-api-access-lch2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:10:23.823951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.823852    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lch2t\" (UniqueName: \"kubernetes.io/projected/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-kube-api-access-lch2t\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:23.823951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:23.823914    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50cb0d2e-89cf-496b-8eea-e1626ddc12cf-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:24.539842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.539813    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"
Nov 03 18:10:24.540077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.539814    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx" event={"ID":"50cb0d2e-89cf-496b-8eea-e1626ddc12cf","Type":"ContainerDied","Data":"883a699a6388f8100e3eca992122e96aaacc37d01423dd14b424f8e9c756ff42"}
Nov 03 18:10:24.540077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.539967    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="883a699a6388f8100e3eca992122e96aaacc37d01423dd14b424f8e9c756ff42"
Nov 03 18:10:24.541668 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.541644    4493 generic.go:356] "Generic (PLEG): container finished" podID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerID="067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658" exitCode=0
Nov 03 18:10:24.541787 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.541698    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" event={"ID":"2b6cd61f-0b32-4936-9c60-158663ebe336","Type":"ContainerDied","Data":"067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658"}
Nov 03 18:10:24.542088 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.542072    4493 scope.go:122] "RemoveContainer" containerID="067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658"
Nov 03 18:10:24.543797 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.543754    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:10:24.543915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.543845    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:10:24.543915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:24.543909    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:10:25.551621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:25.551589    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" event={"ID":"2b6cd61f-0b32-4936-9c60-158663ebe336","Type":"ContainerStarted","Data":"123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae"}
Nov 03 18:10:26.339925 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:26.338958    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:10:26.340708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:26.340675    4493 patch_prober.go:29] interesting pod/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 18:10:26.340846 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:26.340751    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 18:10:27.323910 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:27.323332    4493 patch_prober.go:29] interesting pod/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 18:10:27.323910 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:27.323398    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 18:10:28.337897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:28.337313    4493 patch_prober.go:29] interesting pod/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 18:10:28.337897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:28.337395    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 18:10:29.335899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:29.330327    4493 patch_prober.go:29] interesting pod/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 18:10:29.335899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:29.330404    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 18:10:29.535898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:29.534978    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:10:30.334900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:30.325243    4493 patch_prober.go:29] interesting pod/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 18:10:30.334900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:30.325321    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 18:10:31.322143 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:31.322115    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:10:31.323386 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:31.323364    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:10:36.995260 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:10:36.995220    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:10:37.557631 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:37.557594    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"]
Nov 03 18:10:37.608583 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:37.608550    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-8xsh5"]
Nov 03 18:10:38.713571 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:38.713536    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276f2e0b-545d-416c-b629-9b028aabeeb5" path="/var/lib/kubelet/pods/276f2e0b-545d-416c-b629-9b028aabeeb5/volumes"
Nov 03 18:10:38.825990 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:38.825957    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"]
Nov 03 18:10:38.875772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:38.875741    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c17c6a-tqzfx"]
Nov 03 18:10:40.714482 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:40.714453    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cb0d2e-89cf-496b-8eea-e1626ddc12cf" path="/var/lib/kubelet/pods/50cb0d2e-89cf-496b-8eea-e1626ddc12cf/volumes"
Nov 03 18:10:41.602608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.602571    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"]
Nov 03 18:10:41.603208 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.603191    4493 cpu_manager.go:414] "containerMap: removing container" podUID="50cb0d2e-89cf-496b-8eea-e1626ddc12cf" containerName="revoke-keys"
Nov 03 18:10:41.603259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.603210    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cb0d2e-89cf-496b-8eea-e1626ddc12cf" containerName="revoke-keys"
Nov 03 18:10:41.603319 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.603310    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cb0d2e-89cf-496b-8eea-e1626ddc12cf" containerName="revoke-keys"
Nov 03 18:10:41.606810 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.606794    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.667278 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.667247    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"]
Nov 03 18:10:41.704682 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.704643    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.704851 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.704718    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.704920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.704893    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8d9\" (UniqueName: \"kubernetes.io/projected/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-kube-api-access-4j8d9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.712680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.712657    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:10:41.712809 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.712714    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:10:41.712809 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.712753    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:10:41.712912 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.712820    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:10:41.806141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.806110    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.806786 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.806754    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.806923 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.806826    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.806923 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.806910    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8d9\" (UniqueName: \"kubernetes.io/projected/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-kube-api-access-4j8d9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.809137 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.809118    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.870547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.870488    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8d9\" (UniqueName: \"kubernetes.io/projected/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-kube-api-access-4j8d9\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:41.919476 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:41.919450    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:42.299106 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:42.299075    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"]
Nov 03 18:10:42.300073 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:10:42.300044    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e62f3ec_fa1b_4e20_bc47_4e296aca74b4.slice/crio-3b15a849cf82de400b74eb418d374e5aefcfd8952f212f931e46277fec3a0b13 WatchSource:0}: Error finding container 3b15a849cf82de400b74eb418d374e5aefcfd8952f212f931e46277fec3a0b13: Status 404 returned error can't find the container with id 3b15a849cf82de400b74eb418d374e5aefcfd8952f212f931e46277fec3a0b13
Nov 03 18:10:42.675470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:42.675437    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9" event={"ID":"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4","Type":"ContainerStarted","Data":"0681e025b01e816fcff678805e9b696d0300505706ee68e0aa33075fb484de4c"}
Nov 03 18:10:42.675588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:42.675490    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9" event={"ID":"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4","Type":"ContainerStarted","Data":"3b15a849cf82de400b74eb418d374e5aefcfd8952f212f931e46277fec3a0b13"}
Nov 03 18:10:42.795587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:42.795483    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9" podStartSLOduration=1.7954688559999998 podStartE2EDuration="1.795468856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:42.79485492 +0000 UTC m=+2222.254025908" watchObservedRunningTime="2026-11-03 18:10:42.795468856 +0000 UTC m=+2222.254639842"
Nov 03 18:10:43.683194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:43.683110    4493 generic.go:356] "Generic (PLEG): container finished" podID="3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" containerID="0681e025b01e816fcff678805e9b696d0300505706ee68e0aa33075fb484de4c" exitCode=0
Nov 03 18:10:43.683627 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:43.683249    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9" event={"ID":"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4","Type":"ContainerDied","Data":"0681e025b01e816fcff678805e9b696d0300505706ee68e0aa33075fb484de4c"}
Nov 03 18:10:44.835925 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.835893    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:44.949376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.949281    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-openshift-ca\") pod \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") "
Nov 03 18:10:44.949376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.949344    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-kube-api-access-4j8d9\" (UniqueName: \"kubernetes.io/projected/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-kube-api-access-4j8d9\") pod \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") "
Nov 03 18:10:44.949582 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.949497    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-xks-ca\") pod \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\" (UID: \"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4\") "
Nov 03 18:10:44.949728 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.949701    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-openshift-ca" pod "3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" (UID: "3e62f3ec-fa1b-4e20-bc47-4e296aca74b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:10:44.949972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.949949    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:44.951631 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.951605    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-xks-ca" pod "3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" (UID: "3e62f3ec-fa1b-4e20-bc47-4e296aca74b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:10:44.951631 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:44.951627    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-kube-api-access-4j8d9" pod "3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" (UID: "3e62f3ec-fa1b-4e20-bc47-4e296aca74b4"). InnerVolumeSpecName "kube-api-access-4j8d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:10:45.051014 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:45.050978    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8d9\" (UniqueName: \"kubernetes.io/projected/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-kube-api-access-4j8d9\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:45.051014 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:45.051006    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:45.696669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:45.696635    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"
Nov 03 18:10:45.696669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:45.696654    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9" event={"ID":"3e62f3ec-fa1b-4e20-bc47-4e296aca74b4","Type":"ContainerDied","Data":"3b15a849cf82de400b74eb418d374e5aefcfd8952f212f931e46277fec3a0b13"}
Nov 03 18:10:45.696902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:45.696707    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b15a849cf82de400b74eb418d374e5aefcfd8952f212f931e46277fec3a0b13"
Nov 03 18:10:50.575564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.575524    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"]
Nov 03 18:10:50.576377 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.576356    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" containerName="revoke-keys"
Nov 03 18:10:50.576502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.576380    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" containerName="revoke-keys"
Nov 03 18:10:50.576585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.576552    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" containerName="revoke-keys"
Nov 03 18:10:50.580347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.580325    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.619290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.619248    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"]
Nov 03 18:10:50.624030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.624001    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.633218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.633188    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"]
Nov 03 18:10:50.664965 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.664935    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6ed3e95c-dockercfg-7gwjm\""
Nov 03 18:10:50.665253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.665229    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6ed3e95c\""
Nov 03 18:10:50.679018 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.678993    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:10:50.681093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.681067    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:10:50.711233 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.710101    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"]
Nov 03 18:10:50.721970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.721939    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.726141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.725051    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.726141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.725129    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87362f49-a7bb-45ff-a8d4-2fd9acad9728-plugins-config-volume\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.726141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.725158    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75f2p\" (UniqueName: \"kubernetes.io/projected/67ad4694-6337-4753-bb79-765f3f0d89a2-kube-api-access-75f2p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.726141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.725329    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkc8\" (UniqueName: \"kubernetes.io/projected/87362f49-a7bb-45ff-a8d4-2fd9acad9728-kube-api-access-dlkc8\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.735357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.735318    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:10:50.735691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.735671    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:10:50.742631 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.742599    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"]
Nov 03 18:10:50.746719 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.746696    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:50.806336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.806303    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"]
Nov 03 18:10:50.827210 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.827120    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:50.827436 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.827416    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkc8\" (UniqueName: \"kubernetes.io/projected/87362f49-a7bb-45ff-a8d4-2fd9acad9728-kube-api-access-dlkc8\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.827552 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.827454    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75c2m\" (UniqueName: \"kubernetes.io/projected/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-kube-api-access-75c2m\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:50.827727 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.827707    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.827908 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.827838    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.828082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.827927    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87362f49-a7bb-45ff-a8d4-2fd9acad9728-plugins-config-volume\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.828082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.827957    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75f2p\" (UniqueName: \"kubernetes.io/projected/67ad4694-6337-4753-bb79-765f3f0d89a2-kube-api-access-75f2p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.828824 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.828798    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.829029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.829008    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87362f49-a7bb-45ff-a8d4-2fd9acad9728-plugins-config-volume\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.837336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.837301    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.890945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.890917    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkc8\" (UniqueName: \"kubernetes.io/projected/87362f49-a7bb-45ff-a8d4-2fd9acad9728-kube-api-access-dlkc8\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.891092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.891004    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75f2p\" (UniqueName: \"kubernetes.io/projected/67ad4694-6337-4753-bb79-765f3f0d89a2-kube-api-access-75f2p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.900854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.900829    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:50.929789 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.929761    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:50.929914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.929800    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75c2m\" (UniqueName: \"kubernetes.io/projected/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-kube-api-access-75c2m\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:50.930664 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.930636    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:50.942500 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.942470    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:50.994579 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:50.994511    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75c2m\" (UniqueName: \"kubernetes.io/projected/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-kube-api-access-75c2m\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:51.062084 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.062054    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:51.280372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.280337    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"]
Nov 03 18:10:51.324061 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:10:51.324031    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87362f49_a7bb_45ff_a8d4_2fd9acad9728.slice/crio-1ecbca0d1b5cc3d89c42623fab8d9f079ad6fb8bf5fa0967ce87ddcf7842925a WatchSource:0}: Error finding container 1ecbca0d1b5cc3d89c42623fab8d9f079ad6fb8bf5fa0967ce87ddcf7842925a: Status 404 returned error can't find the container with id 1ecbca0d1b5cc3d89c42623fab8d9f079ad6fb8bf5fa0967ce87ddcf7842925a
Nov 03 18:10:51.336354 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.336326    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"]
Nov 03 18:10:51.338034 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:10:51.338010    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ad4694_6337_4753_bb79_765f3f0d89a2.slice/crio-ea8653b6986056ce75205d7ee960e59835a0381cbf2a4531ccba96511aa2fa6b WatchSource:0}: Error finding container ea8653b6986056ce75205d7ee960e59835a0381cbf2a4531ccba96511aa2fa6b: Status 404 returned error can't find the container with id ea8653b6986056ce75205d7ee960e59835a0381cbf2a4531ccba96511aa2fa6b
Nov 03 18:10:51.447062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.447039    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"]
Nov 03 18:10:51.447609 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:10:51.447581    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f68b88_be7d_42e9_8b64_c94c23c9d67f.slice/crio-b02e3f1c08f3b577236e6cd05c94b77bc3f27afa405f1051acfbcd1526f61dc4 WatchSource:0}: Error finding container b02e3f1c08f3b577236e6cd05c94b77bc3f27afa405f1051acfbcd1526f61dc4: Status 404 returned error can't find the container with id b02e3f1c08f3b577236e6cd05c94b77bc3f27afa405f1051acfbcd1526f61dc4
Nov 03 18:10:51.743999 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.743968    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g" event={"ID":"87362f49-a7bb-45ff-a8d4-2fd9acad9728","Type":"ContainerStarted","Data":"6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b"}
Nov 03 18:10:51.744382 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.744023    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g" event={"ID":"87362f49-a7bb-45ff-a8d4-2fd9acad9728","Type":"ContainerStarted","Data":"1ecbca0d1b5cc3d89c42623fab8d9f079ad6fb8bf5fa0967ce87ddcf7842925a"}
Nov 03 18:10:51.744382 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.744054    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:10:51.745752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.745720    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp" event={"ID":"a3f68b88-be7d-42e9-8b64-c94c23c9d67f","Type":"ContainerStarted","Data":"ddf0fdd09d7e27a41c572530abae23cfaf66843c2dd1cd16129d215aeca04ca7"}
Nov 03 18:10:51.745856 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.745781    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp" event={"ID":"a3f68b88-be7d-42e9-8b64-c94c23c9d67f","Type":"ContainerStarted","Data":"b02e3f1c08f3b577236e6cd05c94b77bc3f27afa405f1051acfbcd1526f61dc4"}
Nov 03 18:10:51.745856 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.745815    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:10:51.747363 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.747340    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x" event={"ID":"67ad4694-6337-4753-bb79-765f3f0d89a2","Type":"ContainerStarted","Data":"5174b0a945e1da60e6ee96a05a0375a8a120b38a748da5055ad036d76fc01856"}
Nov 03 18:10:51.747443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.747381    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x" event={"ID":"67ad4694-6337-4753-bb79-765f3f0d89a2","Type":"ContainerStarted","Data":"ea8653b6986056ce75205d7ee960e59835a0381cbf2a4531ccba96511aa2fa6b"}
Nov 03 18:10:51.863473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.863426    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g" podStartSLOduration=1.863411763 podStartE2EDuration="1.863411763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:51.863199068 +0000 UTC m=+2231.322370066" watchObservedRunningTime="2026-11-03 18:10:51.863411763 +0000 UTC m=+2231.322582748"
Nov 03 18:10:51.985434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:51.985373    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x" podStartSLOduration=1.985359639 podStartE2EDuration="1.985359639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:51.983965824 +0000 UTC m=+2231.443136838" watchObservedRunningTime="2026-11-03 18:10:51.985359639 +0000 UTC m=+2231.444530625"
Nov 03 18:10:52.106663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:52.106554    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp" podStartSLOduration=2.106534917 podStartE2EDuration="2.106534917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:10:52.10269891 +0000 UTC m=+2231.561869897" watchObservedRunningTime="2026-11-03 18:10:52.106534917 +0000 UTC m=+2231.565705906"
Nov 03 18:10:52.754492 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:52.754456    4493 generic.go:356] "Generic (PLEG): container finished" podID="67ad4694-6337-4753-bb79-765f3f0d89a2" containerID="5174b0a945e1da60e6ee96a05a0375a8a120b38a748da5055ad036d76fc01856" exitCode=0
Nov 03 18:10:52.754866 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:52.754540    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x" event={"ID":"67ad4694-6337-4753-bb79-765f3f0d89a2","Type":"ContainerDied","Data":"5174b0a945e1da60e6ee96a05a0375a8a120b38a748da5055ad036d76fc01856"}
Nov 03 18:10:53.903699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:53.903676    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:10:54.069507 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.069426    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ad4694-6337-4753-bb79-765f3f0d89a2-kube-api-access-75f2p\" (UniqueName: \"kubernetes.io/projected/67ad4694-6337-4753-bb79-765f3f0d89a2-kube-api-access-75f2p\") pod \"67ad4694-6337-4753-bb79-765f3f0d89a2\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") "
Nov 03 18:10:54.069507 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.069487    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-openshift-ca\") pod \"67ad4694-6337-4753-bb79-765f3f0d89a2\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") "
Nov 03 18:10:54.069732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.069555    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-xks-ca\") pod \"67ad4694-6337-4753-bb79-765f3f0d89a2\" (UID: \"67ad4694-6337-4753-bb79-765f3f0d89a2\") "
Nov 03 18:10:54.069892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.069843    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-openshift-ca" pod "67ad4694-6337-4753-bb79-765f3f0d89a2" (UID: "67ad4694-6337-4753-bb79-765f3f0d89a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:10:54.070290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.070269    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:54.071768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.071745    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ad4694-6337-4753-bb79-765f3f0d89a2-kube-api-access-75f2p" pod "67ad4694-6337-4753-bb79-765f3f0d89a2" (UID: "67ad4694-6337-4753-bb79-765f3f0d89a2"). InnerVolumeSpecName "kube-api-access-75f2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:10:54.072017 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.072001    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-xks-ca" pod "67ad4694-6337-4753-bb79-765f3f0d89a2" (UID: "67ad4694-6337-4753-bb79-765f3f0d89a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:10:54.171200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.171171    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75f2p\" (UniqueName: \"kubernetes.io/projected/67ad4694-6337-4753-bb79-765f3f0d89a2-kube-api-access-75f2p\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:54.171200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.171198    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67ad4694-6337-4753-bb79-765f3f0d89a2-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:10:54.768036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.768000    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x" event={"ID":"67ad4694-6337-4753-bb79-765f3f0d89a2","Type":"ContainerDied","Data":"ea8653b6986056ce75205d7ee960e59835a0381cbf2a4531ccba96511aa2fa6b"}
Nov 03 18:10:54.768225 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.768056    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea8653b6986056ce75205d7ee960e59835a0381cbf2a4531ccba96511aa2fa6b"
Nov 03 18:10:54.768225 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:10:54.768010    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"
Nov 03 18:11:02.758723 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:02.758691    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:11:02.761113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:02.759816    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:11:06.993162 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:11:06.993120    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:11:13.223658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:13.223621    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"]
Nov 03 18:11:13.269274 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:13.269246    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-ad914d-k8kts"]
Nov 03 18:11:14.518283 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:14.518238    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"]
Nov 03 18:11:14.564854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:14.564819    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d9a300-cwcsv"]
Nov 03 18:11:14.713591 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:14.713552    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd52f69f-ac8e-4143-94e5-4777c9728571" path="/var/lib/kubelet/pods/bd52f69f-ac8e-4143-94e5-4777c9728571/volumes"
Nov 03 18:11:14.714132 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:14.714110    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5eb4ec0-59d5-47a7-a39d-f6fab041c412" path="/var/lib/kubelet/pods/d5eb4ec0-59d5-47a7-a39d-f6fab041c412/volumes"
Nov 03 18:11:16.259470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.259428    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"]
Nov 03 18:11:16.260209 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.260188    4493 cpu_manager.go:414] "containerMap: removing container" podUID="67ad4694-6337-4753-bb79-765f3f0d89a2" containerName="revoke-keys"
Nov 03 18:11:16.260322 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.260211    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ad4694-6337-4753-bb79-765f3f0d89a2" containerName="revoke-keys"
Nov 03 18:11:16.260387 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.260362    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ad4694-6337-4753-bb79-765f3f0d89a2" containerName="revoke-keys"
Nov 03 18:11:16.264844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.264822    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.323846 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.323816    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"]
Nov 03 18:11:16.370441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.370409    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:11:16.370441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.370417    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:11:16.370441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.370438    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:11:16.370676 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.370510    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:11:16.401853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.401820    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.402025 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.402001    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.402080 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.402036    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7psc\" (UniqueName: \"kubernetes.io/projected/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-kube-api-access-h7psc\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.503423 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.503390    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.503576 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.503432    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7psc\" (UniqueName: \"kubernetes.io/projected/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-kube-api-access-h7psc\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.503576 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.503552    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.504169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.504149    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.505903 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.505867    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.566986 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.566921    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7psc\" (UniqueName: \"kubernetes.io/projected/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-kube-api-access-h7psc\") pod \"maas-api-revoke-keys-e2e-api-a-b05459-244f4\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:16.578917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:16.578867    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:17.050384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:17.050360    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"]
Nov 03 18:11:17.948293 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:17.948264    4493 generic.go:356] "Generic (PLEG): container finished" podID="20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" containerID="c806696af4ed4965dab79ddb7e8fc15d184ab85c25d1fa85f7cc84e34db0e25f" exitCode=0
Nov 03 18:11:17.948641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:17.948347    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4" event={"ID":"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d","Type":"ContainerDied","Data":"c806696af4ed4965dab79ddb7e8fc15d184ab85c25d1fa85f7cc84e34db0e25f"}
Nov 03 18:11:17.948641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:17.948403    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4" event={"ID":"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d","Type":"ContainerStarted","Data":"15e728f49c4667b327304e3de71e91f19b34abb103f284e0b4594a5c4f9ee65b"}
Nov 03 18:11:19.096301 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.096279    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:19.252788 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.252758    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-openshift-ca\") pod \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") "
Nov 03 18:11:19.253009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.252814    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-kube-api-access-h7psc\" (UniqueName: \"kubernetes.io/projected/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-kube-api-access-h7psc\") pod \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") "
Nov 03 18:11:19.253009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.252945    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-xks-ca\") pod \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\" (UID: \"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d\") "
Nov 03 18:11:19.255165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.253510    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-openshift-ca" pod "20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" (UID: "20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:19.255165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.254265    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:19.263477 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.263438    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-kube-api-access-h7psc" pod "20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" (UID: "20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d"). InnerVolumeSpecName "kube-api-access-h7psc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:19.270511 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.270482    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-xks-ca" pod "20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" (UID: "20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:19.355710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.355674    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7psc\" (UniqueName: \"kubernetes.io/projected/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-kube-api-access-h7psc\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:19.355710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.355709    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:19.961598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.961570    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"
Nov 03 18:11:19.961773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.961606    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4" event={"ID":"20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d","Type":"ContainerDied","Data":"15e728f49c4667b327304e3de71e91f19b34abb103f284e0b4594a5c4f9ee65b"}
Nov 03 18:11:19.961773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:19.961654    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e728f49c4667b327304e3de71e91f19b34abb103f284e0b4594a5c4f9ee65b"
Nov 03 18:11:21.186603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:21.186567    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"]
Nov 03 18:11:21.187130 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:21.186806    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy" containerID="cri-o://aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18" gracePeriod=30
Nov 03 18:11:22.021628 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.021595    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"]
Nov 03 18:11:22.022196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.022181    4493 cpu_manager.go:414] "containerMap: removing container" podUID="20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" containerName="revoke-keys"
Nov 03 18:11:22.022258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.022199    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" containerName="revoke-keys"
Nov 03 18:11:22.022311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.022302    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" containerName="revoke-keys"
Nov 03 18:11:22.025615 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.025601    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.085021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.084993    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"]
Nov 03 18:11:22.093269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.093235    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.093490 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.093324    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.093490 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.093384    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lq2b\" (UniqueName: \"kubernetes.io/projected/118be6a8-be62-4093-9484-f39d0f314d12-kube-api-access-9lq2b\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.131707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.131676    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:11:22.131976 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.131714    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:11:22.131976 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.131718    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:11:22.132178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.131718    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:11:22.195092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.195056    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lq2b\" (UniqueName: \"kubernetes.io/projected/118be6a8-be62-4093-9484-f39d0f314d12-kube-api-access-9lq2b\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.195546 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.195216    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.195546 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.195261    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.195951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.195927    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.197772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.197737    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.258829 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.258789    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lq2b\" (UniqueName: \"kubernetes.io/projected/118be6a8-be62-4093-9484-f39d0f314d12-kube-api-access-9lq2b\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.338781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.338707    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:22.722690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.722668    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"]
Nov 03 18:11:22.724027 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:11:22.723999    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118be6a8_be62_4093_9484_f39d0f314d12.slice/crio-521ad9396aabd92ecf8c6a46a38a8dd3bf56d6794cd1078f5b958145c2979e55 WatchSource:0}: Error finding container 521ad9396aabd92ecf8c6a46a38a8dd3bf56d6794cd1078f5b958145c2979e55: Status 404 returned error can't find the container with id 521ad9396aabd92ecf8c6a46a38a8dd3bf56d6794cd1078f5b958145c2979e55
Nov 03 18:11:22.990973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:22.990856    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg" event={"ID":"118be6a8-be62-4093-9484-f39d0f314d12","Type":"ContainerStarted","Data":"521ad9396aabd92ecf8c6a46a38a8dd3bf56d6794cd1078f5b958145c2979e55"}
Nov 03 18:11:23.998424 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:23.998393    4493 generic.go:356] "Generic (PLEG): container finished" podID="118be6a8-be62-4093-9484-f39d0f314d12" containerID="fc09b5ef239291a9ce7fee5c40f4111168cb27d99f81dd8182d37197938c6d37" exitCode=0
Nov 03 18:11:23.998813 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:23.998474    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg" event={"ID":"118be6a8-be62-4093-9484-f39d0f314d12","Type":"ContainerDied","Data":"fc09b5ef239291a9ce7fee5c40f4111168cb27d99f81dd8182d37197938c6d37"}
Nov 03 18:11:24.048962 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.048933    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"]
Nov 03 18:11:24.052964 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.052943    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.113894 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.111539    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"]
Nov 03 18:11:24.121767 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.121738    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.121997 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.121971    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dbm\" (UniqueName: \"kubernetes.io/projected/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-kube-api-access-w5dbm\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.122082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.122068    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.222955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.222917    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.223134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.223085    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dbm\" (UniqueName: \"kubernetes.io/projected/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-kube-api-access-w5dbm\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.223195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.223142    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.223752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.223729    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.225357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.225335    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.290903 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.290802    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dbm\" (UniqueName: \"kubernetes.io/projected/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-kube-api-access-w5dbm\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.366690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.366657    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:24.746622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:24.746597    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"]
Nov 03 18:11:24.747266 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:11:24.747235    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9451c32b_fcff_4531_b5f9_f7a734b7e3d0.slice/crio-5c143516b8da0a4fdd22a8ffff286830fdd1cbe060df08d443383608b3894515 WatchSource:0}: Error finding container 5c143516b8da0a4fdd22a8ffff286830fdd1cbe060df08d443383608b3894515: Status 404 returned error can't find the container with id 5c143516b8da0a4fdd22a8ffff286830fdd1cbe060df08d443383608b3894515
Nov 03 18:11:25.005754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.005668    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr" event={"ID":"9451c32b-fcff-4531-b5f9-f7a734b7e3d0","Type":"ContainerStarted","Data":"5c143516b8da0a4fdd22a8ffff286830fdd1cbe060df08d443383608b3894515"}
Nov 03 18:11:25.151988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.151938    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:25.239449 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.239422    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-xks-ca\") pod \"118be6a8-be62-4093-9484-f39d0f314d12\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") "
Nov 03 18:11:25.239596 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.239587    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118be6a8-be62-4093-9484-f39d0f314d12-kube-api-access-9lq2b\" (UniqueName: \"kubernetes.io/projected/118be6a8-be62-4093-9484-f39d0f314d12-kube-api-access-9lq2b\") pod \"118be6a8-be62-4093-9484-f39d0f314d12\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") "
Nov 03 18:11:25.239674 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.239610    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-openshift-ca\") pod \"118be6a8-be62-4093-9484-f39d0f314d12\" (UID: \"118be6a8-be62-4093-9484-f39d0f314d12\") "
Nov 03 18:11:25.240084 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.240054    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-openshift-ca" pod "118be6a8-be62-4093-9484-f39d0f314d12" (UID: "118be6a8-be62-4093-9484-f39d0f314d12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:25.241620 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.241599    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-xks-ca" pod "118be6a8-be62-4093-9484-f39d0f314d12" (UID: "118be6a8-be62-4093-9484-f39d0f314d12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:25.241749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.241725    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118be6a8-be62-4093-9484-f39d0f314d12-kube-api-access-9lq2b" pod "118be6a8-be62-4093-9484-f39d0f314d12" (UID: "118be6a8-be62-4093-9484-f39d0f314d12"). InnerVolumeSpecName "kube-api-access-9lq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:25.341792 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.341716    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lq2b\" (UniqueName: \"kubernetes.io/projected/118be6a8-be62-4093-9484-f39d0f314d12-kube-api-access-9lq2b\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:25.341792 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.341745    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:25.341792 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:25.341757    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118be6a8-be62-4093-9484-f39d0f314d12-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.014280 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.014193    4493 generic.go:356] "Generic (PLEG): container finished" podID="9451c32b-fcff-4531-b5f9-f7a734b7e3d0" containerID="f1126918de3508dc3faaeb9fe67f49de6e6f46d266538537d0e44910f6c1173d" exitCode=0
Nov 03 18:11:26.014699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.014289    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr" event={"ID":"9451c32b-fcff-4531-b5f9-f7a734b7e3d0","Type":"ContainerDied","Data":"f1126918de3508dc3faaeb9fe67f49de6e6f46d266538537d0e44910f6c1173d"}
Nov 03 18:11:26.015598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.015578    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"
Nov 03 18:11:26.015717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.015593    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg" event={"ID":"118be6a8-be62-4093-9484-f39d0f314d12","Type":"ContainerDied","Data":"521ad9396aabd92ecf8c6a46a38a8dd3bf56d6794cd1078f5b958145c2979e55"}
Nov 03 18:11:26.015717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.015642    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="521ad9396aabd92ecf8c6a46a38a8dd3bf56d6794cd1078f5b958145c2979e55"
Nov 03 18:11:26.445121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.445091    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:11:26.561902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.561792    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-data\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.561902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.561894    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-socket\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.561928    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-kube-api-access-p9gxx\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-kube-api-access-p9gxx\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.561967    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-certs\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.561995    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b33821-8e02-409a-a925-380e48ea810c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b33821-8e02-409a-a925-380e48ea810c-istiod-ca-cert\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562056    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-istio-token\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-istio-token\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562071    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-data" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562086    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-credential-socket\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562111    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-socket" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.562189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562158    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94b33821-8e02-409a-a925-380e48ea810c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b33821-8e02-409a-a925-380e48ea810c-istio-podinfo\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562301    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-envoy\") pod \"94b33821-8e02-409a-a925-380e48ea810c\" (UID: \"94b33821-8e02-409a-a925-380e48ea810c\") "
Nov 03 18:11:26.562598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562335    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-certs" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.562598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562397    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-credential-socket" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.562598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562459    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b33821-8e02-409a-a925-380e48ea810c-istiod-ca-cert" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:26.562850 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562830    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.562949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562856    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.562949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562889    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.562949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562904    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b33821-8e02-409a-a925-380e48ea810c-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.562949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.562918    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.564337 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.564310    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-istio-token" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:26.564442 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.564401    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-kube-api-access-p9gxx" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "kube-api-access-p9gxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:26.564936 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.564915    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-envoy" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:11:26.564936 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.564916    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94b33821-8e02-409a-a925-380e48ea810c-istio-podinfo" pod "94b33821-8e02-409a-a925-380e48ea810c" (UID: "94b33821-8e02-409a-a925-380e48ea810c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:11:26.664641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.664599    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9gxx\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-kube-api-access-p9gxx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.664641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.664644    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b33821-8e02-409a-a925-380e48ea810c-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.664831 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.664662    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b33821-8e02-409a-a925-380e48ea810c-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:26.664831 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:26.664677    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b33821-8e02-409a-a925-380e48ea810c-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:27.022673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.022637    4493 generic.go:356] "Generic (PLEG): container finished" podID="94b33821-8e02-409a-a925-380e48ea810c" containerID="aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18" exitCode=0
Nov 03 18:11:27.023199 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.022689    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"
Nov 03 18:11:27.023199 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.022727    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" event={"ID":"94b33821-8e02-409a-a925-380e48ea810c","Type":"ContainerDied","Data":"aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18"}
Nov 03 18:11:27.023199 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.022801    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh" event={"ID":"94b33821-8e02-409a-a925-380e48ea810c","Type":"ContainerDied","Data":"7d8a874acff4a6c7fc9a22e5a2c3469cd407899023db86556a387055d7552df0"}
Nov 03 18:11:27.023199 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.022835    4493 scope.go:122] "RemoveContainer" containerID="aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18"
Nov 03 18:11:27.033169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.033148    4493 scope.go:122] "RemoveContainer" containerID="aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18"
Nov 03 18:11:27.033477 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:11:27.033457    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18\": container with ID starting with aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18 not found: ID does not exist" containerID="aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18"
Nov 03 18:11:27.033561 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.033495    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18"} err="failed to get container status \"aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18\": rpc error: code = NotFound desc = could not find container \"aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18\": container with ID starting with aa79f654cbec91109a17c4590441cac53ffc249250840a8171acd5b7bdb3be18 not found: ID does not exist"
Nov 03 18:11:27.158619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.158597    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:27.195208 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.195181    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"]
Nov 03 18:11:27.247583 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.247557    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-9htlh"]
Nov 03 18:11:27.271808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.271777    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-xks-ca\") pod \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") "
Nov 03 18:11:27.271967 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.271821    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-openshift-ca\") pod \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") "
Nov 03 18:11:27.272017 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.271977    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-kube-api-access-w5dbm\" (UniqueName: \"kubernetes.io/projected/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-kube-api-access-w5dbm\") pod \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\" (UID: \"9451c32b-fcff-4531-b5f9-f7a734b7e3d0\") "
Nov 03 18:11:27.272256 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.272234    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-openshift-ca" pod "9451c32b-fcff-4531-b5f9-f7a734b7e3d0" (UID: "9451c32b-fcff-4531-b5f9-f7a734b7e3d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:27.272559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.272491    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:27.274123 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.274069    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-kube-api-access-w5dbm" pod "9451c32b-fcff-4531-b5f9-f7a734b7e3d0" (UID: "9451c32b-fcff-4531-b5f9-f7a734b7e3d0"). InnerVolumeSpecName "kube-api-access-w5dbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:27.274123 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.274071    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-xks-ca" pod "9451c32b-fcff-4531-b5f9-f7a734b7e3d0" (UID: "9451c32b-fcff-4531-b5f9-f7a734b7e3d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:27.373975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.373939    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:27.373975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:27.373971    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5dbm\" (UniqueName: \"kubernetes.io/projected/9451c32b-fcff-4531-b5f9-f7a734b7e3d0-kube-api-access-w5dbm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:28.031231 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:28.031199    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"
Nov 03 18:11:28.031660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:28.031242    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr" event={"ID":"9451c32b-fcff-4531-b5f9-f7a734b7e3d0","Type":"ContainerDied","Data":"5c143516b8da0a4fdd22a8ffff286830fdd1cbe060df08d443383608b3894515"}
Nov 03 18:11:28.031660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:28.031295    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c143516b8da0a4fdd22a8ffff286830fdd1cbe060df08d443383608b3894515"
Nov 03 18:11:28.713716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:28.713682    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b33821-8e02-409a-a925-380e48ea810c" path="/var/lib/kubelet/pods/94b33821-8e02-409a-a925-380e48ea810c/volumes"
Nov 03 18:11:36.993092 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:11:36.993053    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:11:37.114525 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.114495    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"]
Nov 03 18:11:37.115163 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115138    4493 cpu_manager.go:414] "containerMap: removing container" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy"
Nov 03 18:11:37.115163 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115157    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy"
Nov 03 18:11:37.115316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115189    4493 cpu_manager.go:414] "containerMap: removing container" podUID="118be6a8-be62-4093-9484-f39d0f314d12" containerName="revoke-keys"
Nov 03 18:11:37.115316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115194    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118be6a8-be62-4093-9484-f39d0f314d12" containerName="revoke-keys"
Nov 03 18:11:37.115316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115216    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9451c32b-fcff-4531-b5f9-f7a734b7e3d0" containerName="revoke-keys"
Nov 03 18:11:37.115316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115223    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9451c32b-fcff-4531-b5f9-f7a734b7e3d0" containerName="revoke-keys"
Nov 03 18:11:37.115456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115319    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b33821-8e02-409a-a925-380e48ea810c" containerName="istio-proxy"
Nov 03 18:11:37.115456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115330    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="118be6a8-be62-4093-9484-f39d0f314d12" containerName="revoke-keys"
Nov 03 18:11:37.115456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.115337    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9451c32b-fcff-4531-b5f9-f7a734b7e3d0" containerName="revoke-keys"
Nov 03 18:11:37.117674 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.117657    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.176456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.176427    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"]
Nov 03 18:11:37.220058 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.220023    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfvn\" (UniqueName: \"kubernetes.io/projected/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-kube-api-access-fsfvn\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.220235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.220214    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.220428 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.220407    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.222655 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.222627    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:11:37.222655 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.222633    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:11:37.222807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.222710    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:11:37.222807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.222760    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:11:37.322204 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.322116    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.322374 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.322210    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfvn\" (UniqueName: \"kubernetes.io/projected/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-kube-api-access-fsfvn\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.322374 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.322318    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.322802 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.322784    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.324617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.324596    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.385807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.385778    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfvn\" (UniqueName: \"kubernetes.io/projected/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-kube-api-access-fsfvn\") pod \"maas-api-revoke-keys-e2e-api-a-f5100c-bhctj\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.433374 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.433345    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:37.813336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:37.813303    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"]
Nov 03 18:11:37.815102 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:11:37.815074    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e99ca7_9380_47f4_9395_f9f48b8ad5fc.slice/crio-09bbbe11e742974f5d557ff253934ae264ef468960c3ba9e53e794794cab392b WatchSource:0}: Error finding container 09bbbe11e742974f5d557ff253934ae264ef468960c3ba9e53e794794cab392b: Status 404 returned error can't find the container with id 09bbbe11e742974f5d557ff253934ae264ef468960c3ba9e53e794794cab392b
Nov 03 18:11:38.131730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:38.131657    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj" event={"ID":"47e99ca7-9380-47f4-9395-f9f48b8ad5fc","Type":"ContainerStarted","Data":"09bbbe11e742974f5d557ff253934ae264ef468960c3ba9e53e794794cab392b"}
Nov 03 18:11:38.378445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:38.378415    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"]
Nov 03 18:11:38.389145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:38.389078    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e75517-92r4x"]
Nov 03 18:11:38.713271 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:38.713245    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ad4694-6337-4753-bb79-765f3f0d89a2" path="/var/lib/kubelet/pods/67ad4694-6337-4753-bb79-765f3f0d89a2/volumes"
Nov 03 18:11:39.139211 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:39.139129    4493 generic.go:356] "Generic (PLEG): container finished" podID="47e99ca7-9380-47f4-9395-f9f48b8ad5fc" containerID="c3136a986b04adb8e3090e4408c6345793aafb8b079b507acb4e274c2283b711" exitCode=0
Nov 03 18:11:39.139601 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:39.139210    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj" event={"ID":"47e99ca7-9380-47f4-9395-f9f48b8ad5fc","Type":"ContainerDied","Data":"c3136a986b04adb8e3090e4408c6345793aafb8b079b507acb4e274c2283b711"}
Nov 03 18:11:40.349320 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.349284    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:40.374288 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.374253    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-kube-api-access-fsfvn\" (UniqueName: \"kubernetes.io/projected/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-kube-api-access-fsfvn\") pod \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") "
Nov 03 18:11:40.374443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.374338    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-openshift-ca\") pod \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") "
Nov 03 18:11:40.374443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.374419    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-xks-ca\") pod \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\" (UID: \"47e99ca7-9380-47f4-9395-f9f48b8ad5fc\") "
Nov 03 18:11:40.374735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.374700    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-openshift-ca" pod "47e99ca7-9380-47f4-9395-f9f48b8ad5fc" (UID: "47e99ca7-9380-47f4-9395-f9f48b8ad5fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:40.374940 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.374922    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:40.376626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.376601    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-kube-api-access-fsfvn" pod "47e99ca7-9380-47f4-9395-f9f48b8ad5fc" (UID: "47e99ca7-9380-47f4-9395-f9f48b8ad5fc"). InnerVolumeSpecName "kube-api-access-fsfvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:40.376626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.376614    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-xks-ca" pod "47e99ca7-9380-47f4-9395-f9f48b8ad5fc" (UID: "47e99ca7-9380-47f4-9395-f9f48b8ad5fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:40.475905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.475834    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:40.475905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:40.475865    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsfvn\" (UniqueName: \"kubernetes.io/projected/47e99ca7-9380-47f4-9395-f9f48b8ad5fc-kube-api-access-fsfvn\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:41.183714 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:41.183683    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"
Nov 03 18:11:41.183942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:41.183714    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj" event={"ID":"47e99ca7-9380-47f4-9395-f9f48b8ad5fc","Type":"ContainerDied","Data":"09bbbe11e742974f5d557ff253934ae264ef468960c3ba9e53e794794cab392b"}
Nov 03 18:11:41.183942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:41.183814    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09bbbe11e742974f5d557ff253934ae264ef468960c3ba9e53e794794cab392b"
Nov 03 18:11:42.235536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:42.235501    4493 scope.go:122] "RemoveContainer" containerID="4440e7ecaf45e046f9d017d4e9ab194f966fb019b25c06b01e825d6fbcf69356"
Nov 03 18:11:48.140525 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.140488    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"]
Nov 03 18:11:48.141061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.141045    4493 cpu_manager.go:414] "containerMap: removing container" podUID="47e99ca7-9380-47f4-9395-f9f48b8ad5fc" containerName="revoke-keys"
Nov 03 18:11:48.141128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.141062    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e99ca7-9380-47f4-9395-f9f48b8ad5fc" containerName="revoke-keys"
Nov 03 18:11:48.141169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.141160    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e99ca7-9380-47f4-9395-f9f48b8ad5fc" containerName="revoke-keys"
Nov 03 18:11:48.144256 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.144240    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.192984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.192957    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5kb2\" (UniqueName: \"kubernetes.io/projected/501d2c83-968c-46c1-9b36-95d125f93a78-kube-api-access-z5kb2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.193116 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.193096    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.193178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.193166    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.204235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.204207    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"]
Nov 03 18:11:48.250400 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.250369    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:11:48.250569 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.250454    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:11:48.250569 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.250477    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:11:48.250662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.250567    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:11:48.294623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.294591    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.294770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.294670    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.294770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.294722    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5kb2\" (UniqueName: \"kubernetes.io/projected/501d2c83-968c-46c1-9b36-95d125f93a78-kube-api-access-z5kb2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.295376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.295353    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.297060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.297041    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.361914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.361891    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5kb2\" (UniqueName: \"kubernetes.io/projected/501d2c83-968c-46c1-9b36-95d125f93a78-kube-api-access-z5kb2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.457249 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.457184    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:48.837120 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:48.837095    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"]
Nov 03 18:11:48.837798 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:11:48.837767    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501d2c83_968c_46c1_9b36_95d125f93a78.slice/crio-2f22ed18ad927a0ab5d7e5532e0892f19e83a21b9092ee9ab2cdc89d6001eb31 WatchSource:0}: Error finding container 2f22ed18ad927a0ab5d7e5532e0892f19e83a21b9092ee9ab2cdc89d6001eb31: Status 404 returned error can't find the container with id 2f22ed18ad927a0ab5d7e5532e0892f19e83a21b9092ee9ab2cdc89d6001eb31
Nov 03 18:11:49.239617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:49.239578    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9" event={"ID":"501d2c83-968c-46c1-9b36-95d125f93a78","Type":"ContainerStarted","Data":"4775661a59250b3a5bbdbfdbb1c4fb8be0d59cddfc00119c0caa620adc21f359"}
Nov 03 18:11:49.240061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:49.239680    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9" event={"ID":"501d2c83-968c-46c1-9b36-95d125f93a78","Type":"ContainerStarted","Data":"2f22ed18ad927a0ab5d7e5532e0892f19e83a21b9092ee9ab2cdc89d6001eb31"}
Nov 03 18:11:49.363691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:49.363641    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9" podStartSLOduration=1.363626835 podStartE2EDuration="1.363626835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:49.361038532 +0000 UTC m=+2288.820209521" watchObservedRunningTime="2026-11-03 18:11:49.363626835 +0000 UTC m=+2288.822797820"
Nov 03 18:11:50.081818 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:50.081731    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"]
Nov 03 18:11:50.246839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:50.246805    4493 generic.go:356] "Generic (PLEG): container finished" podID="501d2c83-968c-46c1-9b36-95d125f93a78" containerID="4775661a59250b3a5bbdbfdbb1c4fb8be0d59cddfc00119c0caa620adc21f359" exitCode=0
Nov 03 18:11:50.247252 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:50.246922    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9" event={"ID":"501d2c83-968c-46c1-9b36-95d125f93a78","Type":"ContainerDied","Data":"4775661a59250b3a5bbdbfdbb1c4fb8be0d59cddfc00119c0caa620adc21f359"}
Nov 03 18:11:51.411690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.411661    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:51.535272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.535179    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501d2c83-968c-46c1-9b36-95d125f93a78-kube-api-access-z5kb2\" (UniqueName: \"kubernetes.io/projected/501d2c83-968c-46c1-9b36-95d125f93a78-kube-api-access-z5kb2\") pod \"501d2c83-968c-46c1-9b36-95d125f93a78\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") "
Nov 03 18:11:51.535455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.535357    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-openshift-ca\") pod \"501d2c83-968c-46c1-9b36-95d125f93a78\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") "
Nov 03 18:11:51.535455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.535409    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-xks-ca\") pod \"501d2c83-968c-46c1-9b36-95d125f93a78\" (UID: \"501d2c83-968c-46c1-9b36-95d125f93a78\") "
Nov 03 18:11:51.536742 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.536667    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-openshift-ca" pod "501d2c83-968c-46c1-9b36-95d125f93a78" (UID: "501d2c83-968c-46c1-9b36-95d125f93a78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:51.546129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.546058    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501d2c83-968c-46c1-9b36-95d125f93a78-kube-api-access-z5kb2" pod "501d2c83-968c-46c1-9b36-95d125f93a78" (UID: "501d2c83-968c-46c1-9b36-95d125f93a78"). InnerVolumeSpecName "kube-api-access-z5kb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:51.550346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.550314    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-xks-ca" pod "501d2c83-968c-46c1-9b36-95d125f93a78" (UID: "501d2c83-968c-46c1-9b36-95d125f93a78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:11:51.637621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.637591    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:51.637621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.637619    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501d2c83-968c-46c1-9b36-95d125f93a78-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:51.637841 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:51.637629    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5kb2\" (UniqueName: \"kubernetes.io/projected/501d2c83-968c-46c1-9b36-95d125f93a78-kube-api-access-z5kb2\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:52.058652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:52.058621    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"]
Nov 03 18:11:52.063572 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:52.063545    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820729f2-f78l5"]
Nov 03 18:11:52.260147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:52.260120    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"
Nov 03 18:11:52.260292 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:52.260154    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9" event={"ID":"501d2c83-968c-46c1-9b36-95d125f93a78","Type":"ContainerDied","Data":"2f22ed18ad927a0ab5d7e5532e0892f19e83a21b9092ee9ab2cdc89d6001eb31"}
Nov 03 18:11:52.260292 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:52.260198    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f22ed18ad927a0ab5d7e5532e0892f19e83a21b9092ee9ab2cdc89d6001eb31"
Nov 03 18:11:52.714203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:52.714169    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b702075c-3db6-4c76-ba14-932a63ba15de" path="/var/lib/kubelet/pods/b702075c-3db6-4c76-ba14-932a63ba15de/volumes"
Nov 03 18:11:55.082294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.082253    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9" podUID="453bcafe-e700-4328-aa65-f956dfad21ce" containerName="payload-pre-processing" containerID="cri-o://57b8e8eb6a73040d038dbd9bd63be4a6d194be794e6a1f4b9c5b8501f00872c6" gracePeriod=25
Nov 03 18:11:55.289560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.289523    4493 generic.go:356] "Generic (PLEG): container finished" podID="453bcafe-e700-4328-aa65-f956dfad21ce" containerID="57b8e8eb6a73040d038dbd9bd63be4a6d194be794e6a1f4b9c5b8501f00872c6" exitCode=0
Nov 03 18:11:55.289738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.289700    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9" event={"ID":"453bcafe-e700-4328-aa65-f956dfad21ce","Type":"ContainerDied","Data":"57b8e8eb6a73040d038dbd9bd63be4a6d194be794e6a1f4b9c5b8501f00872c6"}
Nov 03 18:11:55.336485 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.336431    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:11:55.486760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.486717    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453bcafe-e700-4328-aa65-f956dfad21ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453bcafe-e700-4328-aa65-f956dfad21ce-plugins-config-volume\") pod \"453bcafe-e700-4328-aa65-f956dfad21ce\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") "
Nov 03 18:11:55.486964 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.486769    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453bcafe-e700-4328-aa65-f956dfad21ce-kube-api-access-x855x\" (UniqueName: \"kubernetes.io/projected/453bcafe-e700-4328-aa65-f956dfad21ce-kube-api-access-x855x\") pod \"453bcafe-e700-4328-aa65-f956dfad21ce\" (UID: \"453bcafe-e700-4328-aa65-f956dfad21ce\") "
Nov 03 18:11:55.487187 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.487160    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453bcafe-e700-4328-aa65-f956dfad21ce-plugins-config-volume" pod "453bcafe-e700-4328-aa65-f956dfad21ce" (UID: "453bcafe-e700-4328-aa65-f956dfad21ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:11:55.489140 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.489115    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453bcafe-e700-4328-aa65-f956dfad21ce-kube-api-access-x855x" pod "453bcafe-e700-4328-aa65-f956dfad21ce" (UID: "453bcafe-e700-4328-aa65-f956dfad21ce"). InnerVolumeSpecName "kube-api-access-x855x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:11:55.588690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.588607    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x855x\" (UniqueName: \"kubernetes.io/projected/453bcafe-e700-4328-aa65-f956dfad21ce-kube-api-access-x855x\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:55.588690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:55.588637    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/453bcafe-e700-4328-aa65-f956dfad21ce-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:11:56.298945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.298908    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9" event={"ID":"453bcafe-e700-4328-aa65-f956dfad21ce","Type":"ContainerDied","Data":"1d91dd3482ece47d3e83d35a29bfd881366a2af914a76be1c28ea8eb25ab14d0"}
Nov 03 18:11:56.298945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.298932    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"
Nov 03 18:11:56.299457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.299004    4493 scope.go:122] "RemoveContainer" containerID="57b8e8eb6a73040d038dbd9bd63be4a6d194be794e6a1f4b9c5b8501f00872c6"
Nov 03 18:11:56.327939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.327904    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"]
Nov 03 18:11:56.376575 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.376542    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f5100c-bhctj"]
Nov 03 18:11:56.491320 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.491284    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"]
Nov 03 18:11:56.494163 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.494137    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f5100c-7cdb4f6d9f-s6lf9"]
Nov 03 18:11:56.713249 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.713224    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453bcafe-e700-4328-aa65-f956dfad21ce" path="/var/lib/kubelet/pods/453bcafe-e700-4328-aa65-f956dfad21ce/volumes"
Nov 03 18:11:56.713688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:11:56.713674    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e99ca7-9380-47f4-9395-f9f48b8ad5fc" path="/var/lib/kubelet/pods/47e99ca7-9380-47f4-9395-f9f48b8ad5fc/volumes"
Nov 03 18:12:01.301642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:01.301605    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"]
Nov 03 18:12:01.302059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:01.301845    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" containerID="cri-o://75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6" gracePeriod=30
Nov 03 18:12:02.475011 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:02.474975    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:12:02.475425 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:02.475037    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:12:06.575040 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.574983    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 18:12:06.649464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649428    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-credential-socket\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.649632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649484    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-envoy\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.649632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649545    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istiod-ca-cert\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.649717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649652    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-podinfo\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.649717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649686    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-credential-socket" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:06.649717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649705    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-socket\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.649903 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649754    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-kube-api-access-xp7rr\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-kube-api-access-xp7rr\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.649903 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649787    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-token\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.650030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649897    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-socket" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:06.650030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.649913    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-data\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.650030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.650007    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istiod-ca-cert" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:06.650196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.650014    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-certs\") pod \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\" (UID: \"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c\") "
Nov 03 18:12:06.650196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.650169    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-data" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:06.650308 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.650214    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-certs" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:06.651050 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.651022    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.651195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.651052    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.651195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.651067    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.651195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.651082    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.651195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.651096    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.652122 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.652089    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-envoy" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:12:06.652272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.652127    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-kube-api-access-xp7rr" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "kube-api-access-xp7rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:06.652272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.652150    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-podinfo" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:12:06.652272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.652177    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-token" pod "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" (UID: "c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:06.752551 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.752515    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.752551 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.752550    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.752798 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.752564    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp7rr\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-kube-api-access-xp7rr\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.752798 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:06.752581    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:06.997194 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:12:06.997147    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:12:07.382296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.382198    4493 generic.go:356] "Generic (PLEG): container finished" podID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerID="75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6" exitCode=0
Nov 03 18:12:07.382296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.382267    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"
Nov 03 18:12:07.382296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.382281    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" event={"ID":"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c","Type":"ContainerDied","Data":"75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6"}
Nov 03 18:12:07.382790 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.382351    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7" event={"ID":"c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c","Type":"ContainerDied","Data":"dc0c434b0a4b9f876cbba6261ab496021571717130e1cfa0526da552793e2251"}
Nov 03 18:12:07.382790 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.382380    4493 scope.go:122] "RemoveContainer" containerID="75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6"
Nov 03 18:12:07.394062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.394030    4493 scope.go:122] "RemoveContainer" containerID="75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6"
Nov 03 18:12:07.394438 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:12:07.394401    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6\": container with ID starting with 75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6 not found: ID does not exist" containerID="75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6"
Nov 03 18:12:07.394547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.394446    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6"} err="failed to get container status \"75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6\": rpc error: code = NotFound desc = could not find container \"75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6\": container with ID starting with 75f883f1262d4915596de727e50e447e3a7818e47e23236fc6689d919dddccb6 not found: ID does not exist"
Nov 03 18:12:07.559699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.559658    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"]
Nov 03 18:12:07.621895 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:07.616951    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-txlv7"]
Nov 03 18:12:08.713162 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:08.713127    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" path="/var/lib/kubelet/pods/c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c/volumes"
Nov 03 18:12:19.896571 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:19.896534    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"]
Nov 03 18:12:19.947193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:19.947160    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8dfc64-rjzd9"]
Nov 03 18:12:20.713551 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:20.713521    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e62f3ec-fa1b-4e20-bc47-4e296aca74b4" path="/var/lib/kubelet/pods/3e62f3ec-fa1b-4e20-bc47-4e296aca74b4/volumes"
Nov 03 18:12:29.778234 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.778187    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"]
Nov 03 18:12:29.779022 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779000    4493 cpu_manager.go:414] "containerMap: removing container" podUID="501d2c83-968c-46c1-9b36-95d125f93a78" containerName="revoke-keys"
Nov 03 18:12:29.779127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779023    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501d2c83-968c-46c1-9b36-95d125f93a78" containerName="revoke-keys"
Nov 03 18:12:29.779127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779049    4493 cpu_manager.go:414] "containerMap: removing container" podUID="453bcafe-e700-4328-aa65-f956dfad21ce" containerName="payload-pre-processing"
Nov 03 18:12:29.779127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779057    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453bcafe-e700-4328-aa65-f956dfad21ce" containerName="payload-pre-processing"
Nov 03 18:12:29.779127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779086    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy"
Nov 03 18:12:29.779127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779093    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy"
Nov 03 18:12:29.779411 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779281    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="501d2c83-968c-46c1-9b36-95d125f93a78" containerName="revoke-keys"
Nov 03 18:12:29.779411 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779296    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="453bcafe-e700-4328-aa65-f956dfad21ce" containerName="payload-pre-processing"
Nov 03 18:12:29.779411 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.779309    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40c3dc2-b5ce-4570-b0ab-4d441aa89e8c" containerName="istio-proxy"
Nov 03 18:12:29.784440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.784416    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.839350 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.839316    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:12:29.840136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.840111    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"]
Nov 03 18:12:29.872306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.872272    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.872473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.872416    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvpf\" (UniqueName: \"kubernetes.io/projected/8c34e778-8515-46c0-b54f-4cc2e408c82a-kube-api-access-4rvpf\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.872536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.872505    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.886989 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.886952    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:12:29.886989 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.886952    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:12:29.886989 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.886956    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:12:29.974214 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.974180    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.974380 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.974290    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvpf\" (UniqueName: \"kubernetes.io/projected/8c34e778-8515-46c0-b54f-4cc2e408c82a-kube-api-access-4rvpf\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.974380 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.974343    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.975068 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.975039    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:29.977072 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:29.977039    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:30.039064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:30.038965    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvpf\" (UniqueName: \"kubernetes.io/projected/8c34e778-8515-46c0-b54f-4cc2e408c82a-kube-api-access-4rvpf\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:30.101536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:30.101508    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:30.505200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:30.505161    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"]
Nov 03 18:12:30.506002 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:12:30.505972    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c34e778_8515_46c0_b54f_4cc2e408c82a.slice/crio-20b729e46793b0fceacc83e6a66c0b927119a6ae7f64e8728031726768a3c645 WatchSource:0}: Error finding container 20b729e46793b0fceacc83e6a66c0b927119a6ae7f64e8728031726768a3c645: Status 404 returned error can't find the container with id 20b729e46793b0fceacc83e6a66c0b927119a6ae7f64e8728031726768a3c645
Nov 03 18:12:30.534598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:30.534559    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768" event={"ID":"8c34e778-8515-46c0-b54f-4cc2e408c82a","Type":"ContainerStarted","Data":"20b729e46793b0fceacc83e6a66c0b927119a6ae7f64e8728031726768a3c645"}
Nov 03 18:12:31.542672 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:31.542639    4493 generic.go:356] "Generic (PLEG): container finished" podID="8c34e778-8515-46c0-b54f-4cc2e408c82a" containerID="d330059af3a2a5caeb1880501817c1b77db0149632263a20a3594c4408352515" exitCode=0
Nov 03 18:12:31.543111 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:31.542731    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768" event={"ID":"8c34e778-8515-46c0-b54f-4cc2e408c82a","Type":"ContainerDied","Data":"d330059af3a2a5caeb1880501817c1b77db0149632263a20a3594c4408352515"}
Nov 03 18:12:32.692595 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.692568    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:32.821642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.821557    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-openshift-ca\") pod \"8c34e778-8515-46c0-b54f-4cc2e408c82a\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") "
Nov 03 18:12:32.821642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.821604    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c34e778-8515-46c0-b54f-4cc2e408c82a-kube-api-access-4rvpf\" (UniqueName: \"kubernetes.io/projected/8c34e778-8515-46c0-b54f-4cc2e408c82a-kube-api-access-4rvpf\") pod \"8c34e778-8515-46c0-b54f-4cc2e408c82a\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") "
Nov 03 18:12:32.821895 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.821716    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-xks-ca\") pod \"8c34e778-8515-46c0-b54f-4cc2e408c82a\" (UID: \"8c34e778-8515-46c0-b54f-4cc2e408c82a\") "
Nov 03 18:12:32.821967 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.821945    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-openshift-ca" pod "8c34e778-8515-46c0-b54f-4cc2e408c82a" (UID: "8c34e778-8515-46c0-b54f-4cc2e408c82a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:12:32.822957 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.822927    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:32.824464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.824439    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-xks-ca" pod "8c34e778-8515-46c0-b54f-4cc2e408c82a" (UID: "8c34e778-8515-46c0-b54f-4cc2e408c82a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:12:32.827652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.827629    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c34e778-8515-46c0-b54f-4cc2e408c82a-kube-api-access-4rvpf" pod "8c34e778-8515-46c0-b54f-4cc2e408c82a" (UID: "8c34e778-8515-46c0-b54f-4cc2e408c82a"). InnerVolumeSpecName "kube-api-access-4rvpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:12:32.924026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.923993    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c34e778-8515-46c0-b54f-4cc2e408c82a-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:32.924026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:32.924025    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rvpf\" (UniqueName: \"kubernetes.io/projected/8c34e778-8515-46c0-b54f-4cc2e408c82a-kube-api-access-4rvpf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:12:33.556337 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:33.556308    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"
Nov 03 18:12:33.556489 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:33.556303    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768" event={"ID":"8c34e778-8515-46c0-b54f-4cc2e408c82a","Type":"ContainerDied","Data":"20b729e46793b0fceacc83e6a66c0b927119a6ae7f64e8728031726768a3c645"}
Nov 03 18:12:33.556489 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:33.556435    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b729e46793b0fceacc83e6a66c0b927119a6ae7f64e8728031726768a3c645"
Nov 03 18:12:36.992709 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:12:36.992671    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:12:42.395793 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:42.395748    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"]
Nov 03 18:12:42.434055 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:42.434023    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-2569c8-8pgxg"]
Nov 03 18:12:42.712550 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:42.712522    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118be6a8-be62-4093-9484-f39d0f314d12" path="/var/lib/kubelet/pods/118be6a8-be62-4093-9484-f39d0f314d12/volumes"
Nov 03 18:12:43.815526 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:43.815487    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"]
Nov 03 18:12:43.866053 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:43.866022    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-19d275-gfsfr"]
Nov 03 18:12:44.713954 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:44.713909    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9451c32b-fcff-4531-b5f9-f7a734b7e3d0" path="/var/lib/kubelet/pods/9451c32b-fcff-4531-b5f9-f7a734b7e3d0/volumes"
Nov 03 18:12:49.178261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.178230    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"]
Nov 03 18:12:49.178768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.178754    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8c34e778-8515-46c0-b54f-4cc2e408c82a" containerName="revoke-keys"
Nov 03 18:12:49.178813 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.178769    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c34e778-8515-46c0-b54f-4cc2e408c82a" containerName="revoke-keys"
Nov 03 18:12:49.178913 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.178903    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c34e778-8515-46c0-b54f-4cc2e408c82a" containerName="revoke-keys"
Nov 03 18:12:49.183691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.183673    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.239049 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.239020    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6ed3e95c-openshift-default-dockercfg-rk4jm\""
Nov 03 18:12:49.243854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.243825    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhmf\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-kube-api-access-6nhmf\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.243886    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.243995    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f9f592-5d8b-4cd3-9883-aef79b463eca-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.244026    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.244061    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244219 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.244203    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.244245    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.244267    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.244360 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.244347    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.292296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.292266    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"]
Nov 03 18:12:49.345109 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345074    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345109 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345118    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345147    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345214    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345376 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345342    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhmf\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-kube-api-access-6nhmf\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345390    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345431    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f9f592-5d8b-4cd3-9883-aef79b463eca-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345459    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345484    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345539    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345599    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345621    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.345969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.345798    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.346403 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.346384    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f9f592-5d8b-4cd3-9883-aef79b463eca-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.347753 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.347727    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.348049 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.348030    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.409231 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.409204    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.409384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.409307    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhmf\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-kube-api-access-6nhmf\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.496659 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.496619    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:49.930715 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.930676    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"]
Nov 03 18:12:49.931285 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:12:49.931261    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f9f592_5d8b_4cd3_9883_aef79b463eca.slice/crio-2c29e840de0b18b71da18521c7680ef7a86f7d028e7c038f6632b19bae60716d WatchSource:0}: Error finding container 2c29e840de0b18b71da18521c7680ef7a86f7d028e7c038f6632b19bae60716d: Status 404 returned error can't find the container with id 2c29e840de0b18b71da18521c7680ef7a86f7d028e7c038f6632b19bae60716d
Nov 03 18:12:49.933919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.933854    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:12:49.933990 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.933960    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:12:49.934030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:49.933991    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:12:50.664701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:50.664662    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" event={"ID":"57f9f592-5d8b-4cd3-9883-aef79b463eca","Type":"ContainerStarted","Data":"683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1"}
Nov 03 18:12:50.665281 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:50.665254    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" event={"ID":"57f9f592-5d8b-4cd3-9883-aef79b463eca","Type":"ContainerStarted","Data":"2c29e840de0b18b71da18521c7680ef7a86f7d028e7c038f6632b19bae60716d"}
Nov 03 18:12:50.807163 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:50.807112    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podStartSLOduration=1.807097381 podStartE2EDuration="1.807097381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:50.805546175 +0000 UTC m=+2350.264717163" watchObservedRunningTime="2026-11-03 18:12:50.807097381 +0000 UTC m=+2350.266268436"
Nov 03 18:12:51.497670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:51.497640    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:51.499261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:51.499233    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 18:12:51.499392 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:51.499285    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 18:12:52.497751 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:52.497718    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 18:12:52.498185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:52.497772    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 18:12:53.497799 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:53.497755    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 18:12:53.498225 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:53.497825    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 18:12:54.497749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:54.497705    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 18:12:54.497953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:54.497775    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 18:12:55.497396 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:55.497352    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 18:12:55.497594 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:55.497426    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 18:12:56.501781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:56.501752    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:56.502213 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:56.502196    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:12:56.502979 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:12:56.502956    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:13:01.792978 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:01.792944    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"]
Nov 03 18:13:01.843239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:01.843209    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-62aead-j7nf9"]
Nov 03 18:13:02.713318 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:02.713283    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501d2c83-968c-46c1-9b36-95d125f93a78" path="/var/lib/kubelet/pods/501d2c83-968c-46c1-9b36-95d125f93a78/volumes"
Nov 03 18:13:04.305056 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.305025    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"]
Nov 03 18:13:04.309773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.309754    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370323    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370385    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370544 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370508    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370597 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370581    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be7d18-9942-4d43-889a-0cfe69b59bb1-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370639 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370616    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370639 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370633    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370733    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370788 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370765    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.370822 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.370800    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dld\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-kube-api-access-z8dld\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.419090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.419058    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"]
Nov 03 18:13:04.472118 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472084    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472166    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be7d18-9942-4d43-889a-0cfe69b59bb1-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472189    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472207    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472246    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472275    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472325    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dld\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-kube-api-access-z8dld\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472367    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472420    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.472588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472560    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.473201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472607    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.473201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472908    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.473201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472921    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.473201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.472970    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be7d18-9942-4d43-889a-0cfe69b59bb1-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.474922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.474902    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.475263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.475247    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.535986 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.535960    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dld\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-kube-api-access-z8dld\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.535986 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.535974    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:04.623973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:04.623857    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:05.060274 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:05.060239    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"]
Nov 03 18:13:05.060816 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:13:05.060782    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07be7d18_9942_4d43_889a_0cfe69b59bb1.slice/crio-6681e97c305dfad8bd49914a3fd6a9dafb0dcecfdcfe2b549c1a210f07c64c75 WatchSource:0}: Error finding container 6681e97c305dfad8bd49914a3fd6a9dafb0dcecfdcfe2b549c1a210f07c64c75: Status 404 returned error can't find the container with id 6681e97c305dfad8bd49914a3fd6a9dafb0dcecfdcfe2b549c1a210f07c64c75
Nov 03 18:13:05.063325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:05.063291    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:13:05.063397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:05.063355    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:13:05.063397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:05.063382    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:13:05.771465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:05.771425    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" event={"ID":"07be7d18-9942-4d43-889a-0cfe69b59bb1","Type":"ContainerStarted","Data":"18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10"}
Nov 03 18:13:05.771837 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:05.771494    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" event={"ID":"07be7d18-9942-4d43-889a-0cfe69b59bb1","Type":"ContainerStarted","Data":"6681e97c305dfad8bd49914a3fd6a9dafb0dcecfdcfe2b549c1a210f07c64c75"}
Nov 03 18:13:05.947214 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:05.947156    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podStartSLOduration=1.947143336 podStartE2EDuration="1.947143336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:05.946920725 +0000 UTC m=+2365.406091711" watchObservedRunningTime="2026-11-03 18:13:05.947143336 +0000 UTC m=+2365.406314321"
Nov 03 18:13:06.638839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:06.636352    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:06.639087 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:06.638982    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 18:13:06.639087 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:06.639047    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 18:13:07.002079 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:13:07.002035    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:13:07.626341 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:07.626297    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 18:13:07.626975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:07.626942    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 18:13:08.624336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:08.624293    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 18:13:08.624958 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:08.624928    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 18:13:09.631899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:09.627291    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 18:13:09.631899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:09.627368    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 18:13:10.625773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:10.625256    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 18:13:10.625773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:10.625332    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 18:13:11.716714 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.716669    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:11.717222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.717176    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:11.718156 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.718135    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:13:11.871920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.871886    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"]
Nov 03 18:13:11.875954 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.875937    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:11.934937 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.934907    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"]
Nov 03 18:13:11.942160 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.942129    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:11.942321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.942218    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:11.942321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.942286    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:11.942433 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:11.942367    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:13:12.021398 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.021295    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfhn\" (UniqueName: \"kubernetes.io/projected/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-kube-api-access-2bfhn\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.021398 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.021355    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.021666 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.021608    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.127461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.127422    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfhn\" (UniqueName: \"kubernetes.io/projected/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-kube-api-access-2bfhn\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.127774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.127751    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.128098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.128075    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.129183 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.129159    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.152327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.152279    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.194277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.194235    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfhn\" (UniqueName: \"kubernetes.io/projected/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-kube-api-access-2bfhn\") pod \"maas-api-revoke-keys-e2e-worker-w0-07f297-547rv\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.194496 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.194474    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:12.582392 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.582358    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"]
Nov 03 18:13:12.584073 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:13:12.584047    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e620dc5_37e8_4dbc_823b_9d4b967bbb6c.slice/crio-438f3e2be9e131e2a14bebb88bc1d8f47cadfbf717fa85a4d9fce92350196cf7 WatchSource:0}: Error finding container 438f3e2be9e131e2a14bebb88bc1d8f47cadfbf717fa85a4d9fce92350196cf7: Status 404 returned error can't find the container with id 438f3e2be9e131e2a14bebb88bc1d8f47cadfbf717fa85a4d9fce92350196cf7
Nov 03 18:13:12.820598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:12.820559    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv" event={"ID":"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c","Type":"ContainerStarted","Data":"438f3e2be9e131e2a14bebb88bc1d8f47cadfbf717fa85a4d9fce92350196cf7"}
Nov 03 18:13:13.830626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:13.830539    4493 generic.go:356] "Generic (PLEG): container finished" podID="3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" containerID="20e19a4868045dda385fa6315f681d9335bfa36b9819c817c943ffb6dc732cb0" exitCode=0
Nov 03 18:13:13.830626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:13.830589    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv" event={"ID":"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c","Type":"ContainerDied","Data":"20e19a4868045dda385fa6315f681d9335bfa36b9819c817c943ffb6dc732cb0"}
Nov 03 18:13:14.990992 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:14.990959    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:15.070188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.070151    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-kube-api-access-2bfhn\" (UniqueName: \"kubernetes.io/projected/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-kube-api-access-2bfhn\") pod \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") "
Nov 03 18:13:15.070363 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.070293    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-openshift-ca\") pod \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") "
Nov 03 18:13:15.070363 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.070338    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-xks-ca\") pod \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\" (UID: \"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c\") "
Nov 03 18:13:15.070700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.070674    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-openshift-ca" pod "3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" (UID: "3e620dc5-37e8-4dbc-823b-9d4b967bbb6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:15.071005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.070984    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:15.072467 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.072441    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-kube-api-access-2bfhn" pod "3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" (UID: "3e620dc5-37e8-4dbc-823b-9d4b967bbb6c"). InnerVolumeSpecName "kube-api-access-2bfhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:15.072600 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.072529    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-xks-ca" pod "3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" (UID: "3e620dc5-37e8-4dbc-823b-9d4b967bbb6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:15.171911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.171820    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bfhn\" (UniqueName: \"kubernetes.io/projected/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-kube-api-access-2bfhn\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:15.171911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.171854    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:15.845564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.845537    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"
Nov 03 18:13:15.845766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.845567    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv" event={"ID":"3e620dc5-37e8-4dbc-823b-9d4b967bbb6c","Type":"ContainerDied","Data":"438f3e2be9e131e2a14bebb88bc1d8f47cadfbf717fa85a4d9fce92350196cf7"}
Nov 03 18:13:15.845766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:15.845616    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="438f3e2be9e131e2a14bebb88bc1d8f47cadfbf717fa85a4d9fce92350196cf7"
Nov 03 18:13:16.284319 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:16.284273    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"]
Nov 03 18:13:27.301839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.301756    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"]
Nov 03 18:13:27.302334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.302318    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" containerName="revoke-keys"
Nov 03 18:13:27.302379 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.302336    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" containerName="revoke-keys"
Nov 03 18:13:27.302470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.302445    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" containerName="revoke-keys"
Nov 03 18:13:27.305913 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.305865    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.356379 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.356347    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"]
Nov 03 18:13:27.360350 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.360327    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.365810 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.365784    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"]
Nov 03 18:13:27.404000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.403938    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:27.404186 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.404011    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:13:27.431654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.431617    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:27.431835 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.431775    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:27.434503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.434472    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"]
Nov 03 18:13:27.451466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.451433    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.451621 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.451586    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.451711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.451695    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.451777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.451752    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tg9g\" (UniqueName: \"kubernetes.io/projected/8efb4c96-865e-4dd6-a7c1-816911f15d76-kube-api-access-9tg9g\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.452005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.451979    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.452094 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.452034    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfsc2\" (UniqueName: \"kubernetes.io/projected/8cace8e1-9202-4eca-a19f-493ed9705514-kube-api-access-qfsc2\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.556046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.554348    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tg9g\" (UniqueName: \"kubernetes.io/projected/8efb4c96-865e-4dd6-a7c1-816911f15d76-kube-api-access-9tg9g\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.556244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.556123    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.556244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.556198    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfsc2\" (UniqueName: \"kubernetes.io/projected/8cace8e1-9202-4eca-a19f-493ed9705514-kube-api-access-qfsc2\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.563891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.556505    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.563891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.556757    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.563891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.556899    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.563891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.557782    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.563891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.558805    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.563891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.562955    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.565067 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.564672    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.621488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.621455    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tg9g\" (UniqueName: \"kubernetes.io/projected/8efb4c96-865e-4dd6-a7c1-816911f15d76-kube-api-access-9tg9g\") pod \"maas-api-revoke-sub-a6c866d2-qbkts\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.622665 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.622638    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfsc2\" (UniqueName: \"kubernetes.io/projected/8cace8e1-9202-4eca-a19f-493ed9705514-kube-api-access-qfsc2\") pod \"maas-api-revoke-sub-3a03c71e-m6zgx\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:27.679752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.679718    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:27.919681 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:27.919599    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:28.084744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.084712    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"]
Nov 03 18:13:28.085542 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:13:28.085514    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efb4c96_865e_4dd6_a7c1_816911f15d76.slice/crio-647979054ebfb117e4044e771b4ffc8da3b6946aecef87ca1a1594a45a429288 WatchSource:0}: Error finding container 647979054ebfb117e4044e771b4ffc8da3b6946aecef87ca1a1594a45a429288: Status 404 returned error can't find the container with id 647979054ebfb117e4044e771b4ffc8da3b6946aecef87ca1a1594a45a429288
Nov 03 18:13:28.299929 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.299900    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"]
Nov 03 18:13:28.301207 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:13:28.301182    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cace8e1_9202_4eca_a19f_493ed9705514.slice/crio-4b989d97af6cb5f5240e92a5a4ab2f797f961f6ac8b2d6f510fc474a2cfce0c7 WatchSource:0}: Error finding container 4b989d97af6cb5f5240e92a5a4ab2f797f961f6ac8b2d6f510fc474a2cfce0c7: Status 404 returned error can't find the container with id 4b989d97af6cb5f5240e92a5a4ab2f797f961f6ac8b2d6f510fc474a2cfce0c7
Nov 03 18:13:28.940246 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.939104    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts" event={"ID":"8efb4c96-865e-4dd6-a7c1-816911f15d76","Type":"ContainerStarted","Data":"1d8e04b3e4460ae5b70bf59d74636b9cf8a343a8a53296e3e55a55516f3c48cf"}
Nov 03 18:13:28.940246 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.939177    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts" event={"ID":"8efb4c96-865e-4dd6-a7c1-816911f15d76","Type":"ContainerStarted","Data":"647979054ebfb117e4044e771b4ffc8da3b6946aecef87ca1a1594a45a429288"}
Nov 03 18:13:28.944667 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.943215    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx" event={"ID":"8cace8e1-9202-4eca-a19f-493ed9705514","Type":"ContainerStarted","Data":"2105fbe9344fb2acc8e35af5802a2ebeeed7bb52394cd69e2349fe075c28581a"}
Nov 03 18:13:28.944667 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.943272    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx" event={"ID":"8cace8e1-9202-4eca-a19f-493ed9705514","Type":"ContainerStarted","Data":"4b989d97af6cb5f5240e92a5a4ab2f797f961f6ac8b2d6f510fc474a2cfce0c7"}
Nov 03 18:13:28.991137 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.991063    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"]
Nov 03 18:13:28.995021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:28.995004    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.061918 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.057039    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"]
Nov 03 18:13:29.064798 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.064749    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts" podStartSLOduration=2.064734147 podStartE2EDuration="2.064734147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:29.062442669 +0000 UTC m=+2388.521613657" watchObservedRunningTime="2026-11-03 18:13:29.064734147 +0000 UTC m=+2388.523905131"
Nov 03 18:13:29.081163 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.081120    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.081346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.081267    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.081440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.081413    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktx6\" (UniqueName: \"kubernetes.io/projected/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-kube-api-access-4ktx6\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.182563 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.182532    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktx6\" (UniqueName: \"kubernetes.io/projected/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-kube-api-access-4ktx6\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.182757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.182672    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.182757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.182739    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.183348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.183315    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.185219 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.185194    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.245980 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.245901    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktx6\" (UniqueName: \"kubernetes.io/projected/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-kube-api-access-4ktx6\") pod \"maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.300079 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.299981    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx" podStartSLOduration=2.299962983 podStartE2EDuration="2.299962983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:29.29988691 +0000 UTC m=+2388.759057958" watchObservedRunningTime="2026-11-03 18:13:29.299962983 +0000 UTC m=+2388.759133971"
Nov 03 18:13:29.309077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.309052    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:29.689966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.689942    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"]
Nov 03 18:13:29.690433 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:13:29.690394    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8d69ff_4876_4f5d_8f2a_4a9daf4080ac.slice/crio-39474b7bb4956ba900bbc577c9b36e483263aa57098b7a1f323c032e2b92a445 WatchSource:0}: Error finding container 39474b7bb4956ba900bbc577c9b36e483263aa57098b7a1f323c032e2b92a445: Status 404 returned error can't find the container with id 39474b7bb4956ba900bbc577c9b36e483263aa57098b7a1f323c032e2b92a445
Nov 03 18:13:29.951340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.951253    4493 generic.go:356] "Generic (PLEG): container finished" podID="8efb4c96-865e-4dd6-a7c1-816911f15d76" containerID="1d8e04b3e4460ae5b70bf59d74636b9cf8a343a8a53296e3e55a55516f3c48cf" exitCode=0
Nov 03 18:13:29.951806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.951337    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts" event={"ID":"8efb4c96-865e-4dd6-a7c1-816911f15d76","Type":"ContainerDied","Data":"1d8e04b3e4460ae5b70bf59d74636b9cf8a343a8a53296e3e55a55516f3c48cf"}
Nov 03 18:13:29.953130 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.953096    4493 generic.go:356] "Generic (PLEG): container finished" podID="8cace8e1-9202-4eca-a19f-493ed9705514" containerID="2105fbe9344fb2acc8e35af5802a2ebeeed7bb52394cd69e2349fe075c28581a" exitCode=0
Nov 03 18:13:29.953252 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.953127    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx" event={"ID":"8cace8e1-9202-4eca-a19f-493ed9705514","Type":"ContainerDied","Data":"2105fbe9344fb2acc8e35af5802a2ebeeed7bb52394cd69e2349fe075c28581a"}
Nov 03 18:13:29.954580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:29.954560    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh" event={"ID":"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac","Type":"ContainerStarted","Data":"39474b7bb4956ba900bbc577c9b36e483263aa57098b7a1f323c032e2b92a445"}
Nov 03 18:13:30.961991 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:30.961956    4493 generic.go:356] "Generic (PLEG): container finished" podID="ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" containerID="f8330a81e8dbab21766de6a3a924149fe84f5997c84ab7a7a7fccc902f433ba2" exitCode=0
Nov 03 18:13:30.962434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:30.962028    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh" event={"ID":"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac","Type":"ContainerDied","Data":"f8330a81e8dbab21766de6a3a924149fe84f5997c84ab7a7a7fccc902f433ba2"}
Nov 03 18:13:31.143399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.143369    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:31.172674 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.172653    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:31.313279 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313206    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-openshift-ca\") pod \"8efb4c96-865e-4dd6-a7c1-816911f15d76\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") "
Nov 03 18:13:31.313279 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313265    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-xks-ca\") pod \"8cace8e1-9202-4eca-a19f-493ed9705514\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") "
Nov 03 18:13:31.313455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313301    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-xks-ca\") pod \"8efb4c96-865e-4dd6-a7c1-816911f15d76\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") "
Nov 03 18:13:31.313455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313375    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efb4c96-865e-4dd6-a7c1-816911f15d76-kube-api-access-9tg9g\" (UniqueName: \"kubernetes.io/projected/8efb4c96-865e-4dd6-a7c1-816911f15d76-kube-api-access-9tg9g\") pod \"8efb4c96-865e-4dd6-a7c1-816911f15d76\" (UID: \"8efb4c96-865e-4dd6-a7c1-816911f15d76\") "
Nov 03 18:13:31.313553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313531    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-openshift-ca\") pod \"8cace8e1-9202-4eca-a19f-493ed9705514\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") "
Nov 03 18:13:31.313600 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313589    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cace8e1-9202-4eca-a19f-493ed9705514-kube-api-access-qfsc2\" (UniqueName: \"kubernetes.io/projected/8cace8e1-9202-4eca-a19f-493ed9705514-kube-api-access-qfsc2\") pod \"8cace8e1-9202-4eca-a19f-493ed9705514\" (UID: \"8cace8e1-9202-4eca-a19f-493ed9705514\") "
Nov 03 18:13:31.313678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313635    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-openshift-ca" pod "8efb4c96-865e-4dd6-a7c1-816911f15d76" (UID: "8efb4c96-865e-4dd6-a7c1-816911f15d76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:31.313955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.313908    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-openshift-ca" pod "8cace8e1-9202-4eca-a19f-493ed9705514" (UID: "8cace8e1-9202-4eca-a19f-493ed9705514"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:31.314607 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.314584    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.314657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.314617    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.315801 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.315773    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-xks-ca" pod "8cace8e1-9202-4eca-a19f-493ed9705514" (UID: "8cace8e1-9202-4eca-a19f-493ed9705514"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:31.315968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.315857    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-xks-ca" pod "8efb4c96-865e-4dd6-a7c1-816911f15d76" (UID: "8efb4c96-865e-4dd6-a7c1-816911f15d76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:31.316249 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.316227    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cace8e1-9202-4eca-a19f-493ed9705514-kube-api-access-qfsc2" pod "8cace8e1-9202-4eca-a19f-493ed9705514" (UID: "8cace8e1-9202-4eca-a19f-493ed9705514"). InnerVolumeSpecName "kube-api-access-qfsc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:31.316311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.316257    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efb4c96-865e-4dd6-a7c1-816911f15d76-kube-api-access-9tg9g" pod "8efb4c96-865e-4dd6-a7c1-816911f15d76" (UID: "8efb4c96-865e-4dd6-a7c1-816911f15d76"). InnerVolumeSpecName "kube-api-access-9tg9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:31.316566 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.316543    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main" containerID="cri-o://f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995" gracePeriod=45
Nov 03 18:13:31.415705 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.415672    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cace8e1-9202-4eca-a19f-493ed9705514-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.415705 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.415701    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb4c96-865e-4dd6-a7c1-816911f15d76-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.415705 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.415710    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tg9g\" (UniqueName: \"kubernetes.io/projected/8efb4c96-865e-4dd6-a7c1-816911f15d76-kube-api-access-9tg9g\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.415995 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.415720    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfsc2\" (UniqueName: \"kubernetes.io/projected/8cace8e1-9202-4eca-a19f-493ed9705514-kube-api-access-qfsc2\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.571338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.571278    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 18:13:31.718431 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.718402    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63387678-dbfd-43c5-87a1-ef0b581346af-tls-certs\" (UniqueName: \"kubernetes.io/secret/63387678-dbfd-43c5-87a1-ef0b581346af-tls-certs\") pod \"63387678-dbfd-43c5-87a1-ef0b581346af\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") "
Nov 03 18:13:31.718598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.718558    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63387678-dbfd-43c5-87a1-ef0b581346af-kube-api-access-sfv5t\" (UniqueName: \"kubernetes.io/projected/63387678-dbfd-43c5-87a1-ef0b581346af-kube-api-access-sfv5t\") pod \"63387678-dbfd-43c5-87a1-ef0b581346af\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") "
Nov 03 18:13:31.718663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.718652    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-model-cache\") pod \"63387678-dbfd-43c5-87a1-ef0b581346af\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") "
Nov 03 18:13:31.718724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.718699    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-dshm\") pod \"63387678-dbfd-43c5-87a1-ef0b581346af\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") "
Nov 03 18:13:31.719061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.718824    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-home\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-home\") pod \"63387678-dbfd-43c5-87a1-ef0b581346af\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") "
Nov 03 18:13:31.719061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.718889    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-tmp-dir\") pod \"63387678-dbfd-43c5-87a1-ef0b581346af\" (UID: \"63387678-dbfd-43c5-87a1-ef0b581346af\") "
Nov 03 18:13:31.719061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.719012    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-model-cache" pod "63387678-dbfd-43c5-87a1-ef0b581346af" (UID: "63387678-dbfd-43c5-87a1-ef0b581346af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.719575 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.719333    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-tmp-dir" pod "63387678-dbfd-43c5-87a1-ef0b581346af" (UID: "63387678-dbfd-43c5-87a1-ef0b581346af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.719575 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.719403    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-home" pod "63387678-dbfd-43c5-87a1-ef0b581346af" (UID: "63387678-dbfd-43c5-87a1-ef0b581346af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.720145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.719976    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.720145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.720006    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.720145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.720021    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.721138 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.721111    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63387678-dbfd-43c5-87a1-ef0b581346af-tls-certs" pod "63387678-dbfd-43c5-87a1-ef0b581346af" (UID: "63387678-dbfd-43c5-87a1-ef0b581346af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:31.721814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.721794    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63387678-dbfd-43c5-87a1-ef0b581346af-kube-api-access-sfv5t" pod "63387678-dbfd-43c5-87a1-ef0b581346af" (UID: "63387678-dbfd-43c5-87a1-ef0b581346af"). InnerVolumeSpecName "kube-api-access-sfv5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:31.721905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.721795    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-dshm" pod "63387678-dbfd-43c5-87a1-ef0b581346af" (UID: "63387678-dbfd-43c5-87a1-ef0b581346af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:13:31.821321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.821289    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfv5t\" (UniqueName: \"kubernetes.io/projected/63387678-dbfd-43c5-87a1-ef0b581346af-kube-api-access-sfv5t\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.821321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.821318    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63387678-dbfd-43c5-87a1-ef0b581346af-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.821321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.821329    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63387678-dbfd-43c5-87a1-ef0b581346af-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:31.970599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.970563    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts" event={"ID":"8efb4c96-865e-4dd6-a7c1-816911f15d76","Type":"ContainerDied","Data":"647979054ebfb117e4044e771b4ffc8da3b6946aecef87ca1a1594a45a429288"}
Nov 03 18:13:31.971019 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.970625    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647979054ebfb117e4044e771b4ffc8da3b6946aecef87ca1a1594a45a429288"
Nov 03 18:13:31.971019 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.970577    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"
Nov 03 18:13:31.971861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.971838    4493 generic.go:356] "Generic (PLEG): container finished" podID="63387678-dbfd-43c5-87a1-ef0b581346af" containerID="f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995" exitCode=0
Nov 03 18:13:31.972031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.971867    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"
Nov 03 18:13:31.972031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.971916    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" event={"ID":"63387678-dbfd-43c5-87a1-ef0b581346af","Type":"ContainerDied","Data":"f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995"}
Nov 03 18:13:31.972031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.971954    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47" event={"ID":"63387678-dbfd-43c5-87a1-ef0b581346af","Type":"ContainerDied","Data":"fe04278ec94ab0e3e37493c853e1606fddbf46290805983c89611bac29273fa4"}
Nov 03 18:13:31.972031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.971976    4493 scope.go:122] "RemoveContainer" containerID="f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995"
Nov 03 18:13:31.973607 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.973578    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"
Nov 03 18:13:31.973607 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.973606    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx" event={"ID":"8cace8e1-9202-4eca-a19f-493ed9705514","Type":"ContainerDied","Data":"4b989d97af6cb5f5240e92a5a4ab2f797f961f6ac8b2d6f510fc474a2cfce0c7"}
Nov 03 18:13:31.973830 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.973646    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b989d97af6cb5f5240e92a5a4ab2f797f961f6ac8b2d6f510fc474a2cfce0c7"
Nov 03 18:13:31.988090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.988061    4493 scope.go:122] "RemoveContainer" containerID="f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995"
Nov 03 18:13:31.988545 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:13:31.988511    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995\": container with ID starting with f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995 not found: ID does not exist" containerID="f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995"
Nov 03 18:13:31.988676 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:31.988555    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995"} err="failed to get container status \"f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995\": rpc error: code = NotFound desc = could not find container \"f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995\": container with ID starting with f6ed92225b4c759cb2cc697fc2029ae7ee69282e36287593703ae4eaadb69995 not found: ID does not exist"
Nov 03 18:13:32.153469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.153446    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:32.250754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.250679    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"]
Nov 03 18:13:32.256453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.256426    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/simulator-w0-07f297-kserve-589ff58c44-d5r47"]
Nov 03 18:13:32.326506 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.326475    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-xks-ca\") pod \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") "
Nov 03 18:13:32.326660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.326528    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-kube-api-access-4ktx6\" (UniqueName: \"kubernetes.io/projected/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-kube-api-access-4ktx6\") pod \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") "
Nov 03 18:13:32.326660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.326556    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-openshift-ca\") pod \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\" (UID: \"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac\") "
Nov 03 18:13:32.327069 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.327043    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-openshift-ca" pod "ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" (UID: "ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:32.328683 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.328660    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-xks-ca" pod "ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" (UID: "ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:32.328763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.328742    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-kube-api-access-4ktx6" pod "ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" (UID: "ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac"). InnerVolumeSpecName "kube-api-access-4ktx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:32.428237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.428202    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ktx6\" (UniqueName: \"kubernetes.io/projected/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-kube-api-access-4ktx6\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:32.428237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.428232    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:32.428237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.428244    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:32.713858 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.713833    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" path="/var/lib/kubelet/pods/63387678-dbfd-43c5-87a1-ef0b581346af/volumes"
Nov 03 18:13:32.983461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.983381    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"
Nov 03 18:13:32.983924 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.983375    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh" event={"ID":"ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac","Type":"ContainerDied","Data":"39474b7bb4956ba900bbc577c9b36e483263aa57098b7a1f323c032e2b92a445"}
Nov 03 18:13:32.983924 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:32.983504    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39474b7bb4956ba900bbc577c9b36e483263aa57098b7a1f323c032e2b92a445"
Nov 03 18:13:35.177685 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:35.177654    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"]
Nov 03 18:13:35.228295 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:35.228259    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-528e53-7v768"]
Nov 03 18:13:36.716264 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:36.716229    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c34e778-8515-46c0-b54f-4cc2e408c82a" path="/var/lib/kubelet/pods/8c34e778-8515-46c0-b54f-4cc2e408c82a/volumes"
Nov 03 18:13:36.993671 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:13:36.993586    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:13:40.525307 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:40.525277    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"]
Nov 03 18:13:40.638599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:40.638570    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"]
Nov 03 18:13:41.735129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:41.735091    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"]
Nov 03 18:13:41.739601 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:41.739575    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b05459-244f4"]
Nov 03 18:13:42.290127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:42.290094    4493 scope.go:122] "RemoveContainer" containerID="d444e45d862eb850f763646a7a7c102685f811a34ca4d681ec435eaeafb8c19f"
Nov 03 18:13:42.300060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:42.300033    4493 scope.go:122] "RemoveContainer" containerID="589513ec1bcb91f2d3ceedce73c022c32d94091f4912c2e0dd114b617828a6b3"
Nov 03 18:13:42.713117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:42.713094    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d" path="/var/lib/kubelet/pods/20c7d4f3-f8b7-4e30-9dde-6fe2d81c8a3d/volumes"
Nov 03 18:13:45.526273 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:45.526221    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g" podUID="87362f49-a7bb-45ff-a8d4-2fd9acad9728" containerName="payload-processing" containerID="cri-o://6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b" gracePeriod=25
Nov 03 18:13:45.639292 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:45.639230    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp" podUID="a3f68b88-be7d-42e9-8b64-c94c23c9d67f" containerName="payload-pre-processing" containerID="cri-o://ddf0fdd09d7e27a41c572530abae23cfaf66843c2dd1cd16129d215aeca04ca7" gracePeriod=25
Nov 03 18:13:46.045959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.045817    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:13:46.091011 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.090975    4493 generic.go:356] "Generic (PLEG): container finished" podID="87362f49-a7bb-45ff-a8d4-2fd9acad9728" containerID="6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b" exitCode=0
Nov 03 18:13:46.091193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.091035    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"
Nov 03 18:13:46.091193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.091055    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g" event={"ID":"87362f49-a7bb-45ff-a8d4-2fd9acad9728","Type":"ContainerDied","Data":"6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b"}
Nov 03 18:13:46.091193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.091124    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g" event={"ID":"87362f49-a7bb-45ff-a8d4-2fd9acad9728","Type":"ContainerDied","Data":"1ecbca0d1b5cc3d89c42623fab8d9f079ad6fb8bf5fa0967ce87ddcf7842925a"}
Nov 03 18:13:46.091193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.091154    4493 scope.go:122] "RemoveContainer" containerID="6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b"
Nov 03 18:13:46.093157 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.093133    4493 generic.go:356] "Generic (PLEG): container finished" podID="a3f68b88-be7d-42e9-8b64-c94c23c9d67f" containerID="ddf0fdd09d7e27a41c572530abae23cfaf66843c2dd1cd16129d215aeca04ca7" exitCode=0
Nov 03 18:13:46.093287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.093263    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp" event={"ID":"a3f68b88-be7d-42e9-8b64-c94c23c9d67f","Type":"ContainerDied","Data":"ddf0fdd09d7e27a41c572530abae23cfaf66843c2dd1cd16129d215aeca04ca7"}
Nov 03 18:13:46.093419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.093309    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp" event={"ID":"a3f68b88-be7d-42e9-8b64-c94c23c9d67f","Type":"ContainerDied","Data":"b02e3f1c08f3b577236e6cd05c94b77bc3f27afa405f1051acfbcd1526f61dc4"}
Nov 03 18:13:46.093419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.093326    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02e3f1c08f3b577236e6cd05c94b77bc3f27afa405f1051acfbcd1526f61dc4"
Nov 03 18:13:46.096283 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.096261    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:13:46.102259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.102230    4493 scope.go:122] "RemoveContainer" containerID="6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b"
Nov 03 18:13:46.102531 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:13:46.102511    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b\": container with ID starting with 6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b not found: ID does not exist" containerID="6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b"
Nov 03 18:13:46.102610 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.102540    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b"} err="failed to get container status \"6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b\": rpc error: code = NotFound desc = could not find container \"6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b\": container with ID starting with 6baf87eff9a18da275050a46312f1020329a581d2c5805eed749b3db3adb705b not found: ID does not exist"
Nov 03 18:13:46.157251 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.157221    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87362f49-a7bb-45ff-a8d4-2fd9acad9728-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87362f49-a7bb-45ff-a8d4-2fd9acad9728-plugins-config-volume\") pod \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") "
Nov 03 18:13:46.157456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.157426    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-kube-api-access-75c2m\" (UniqueName: \"kubernetes.io/projected/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-kube-api-access-75c2m\") pod \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") "
Nov 03 18:13:46.157529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.157482    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-plugins-config-volume\") pod \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\" (UID: \"a3f68b88-be7d-42e9-8b64-c94c23c9d67f\") "
Nov 03 18:13:46.157529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.157521    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87362f49-a7bb-45ff-a8d4-2fd9acad9728-kube-api-access-dlkc8\" (UniqueName: \"kubernetes.io/projected/87362f49-a7bb-45ff-a8d4-2fd9acad9728-kube-api-access-dlkc8\") pod \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\" (UID: \"87362f49-a7bb-45ff-a8d4-2fd9acad9728\") "
Nov 03 18:13:46.157696 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.157665    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87362f49-a7bb-45ff-a8d4-2fd9acad9728-plugins-config-volume" pod "87362f49-a7bb-45ff-a8d4-2fd9acad9728" (UID: "87362f49-a7bb-45ff-a8d4-2fd9acad9728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:46.157984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.157955    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-plugins-config-volume" pod "a3f68b88-be7d-42e9-8b64-c94c23c9d67f" (UID: "a3f68b88-be7d-42e9-8b64-c94c23c9d67f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:46.158141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.158022    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87362f49-a7bb-45ff-a8d4-2fd9acad9728-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:46.160095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.160066    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-kube-api-access-75c2m" pod "a3f68b88-be7d-42e9-8b64-c94c23c9d67f" (UID: "a3f68b88-be7d-42e9-8b64-c94c23c9d67f"). InnerVolumeSpecName "kube-api-access-75c2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:46.160634 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.160610    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87362f49-a7bb-45ff-a8d4-2fd9acad9728-kube-api-access-dlkc8" pod "87362f49-a7bb-45ff-a8d4-2fd9acad9728" (UID: "87362f49-a7bb-45ff-a8d4-2fd9acad9728"). InnerVolumeSpecName "kube-api-access-dlkc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:46.259206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.259174    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75c2m\" (UniqueName: \"kubernetes.io/projected/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-kube-api-access-75c2m\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:46.259206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.259203    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f68b88-be7d-42e9-8b64-c94c23c9d67f-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:46.259206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.259213    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkc8\" (UniqueName: \"kubernetes.io/projected/87362f49-a7bb-45ff-a8d4-2fd9acad9728-kube-api-access-dlkc8\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:46.517416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.517378    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"]
Nov 03 18:13:46.524749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.524716    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-64t8g"]
Nov 03 18:13:46.713943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:46.713909    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87362f49-a7bb-45ff-a8d4-2fd9acad9728" path="/var/lib/kubelet/pods/87362f49-a7bb-45ff-a8d4-2fd9acad9728/volumes"
Nov 03 18:13:47.068654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:47.068546    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"]
Nov 03 18:13:47.074863 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:47.074829    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-697aa93a-c7ldb"]
Nov 03 18:13:47.100861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:47.100838    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"
Nov 03 18:13:47.273553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:47.273524    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"]
Nov 03 18:13:47.282220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:47.282191    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-pmvqp"]
Nov 03 18:13:48.712888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:48.712847    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f68b88-be7d-42e9-8b64-c94c23c9d67f" path="/var/lib/kubelet/pods/a3f68b88-be7d-42e9-8b64-c94c23c9d67f/volumes"
Nov 03 18:13:48.713353 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:48.713338    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feee9e96-148a-4a0f-b5e3-ffa480675f04" path="/var/lib/kubelet/pods/feee9e96-148a-4a0f-b5e3-ffa480675f04/volumes"
Nov 03 18:13:51.103255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103217    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"]
Nov 03 18:13:51.103754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103734    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8cace8e1-9202-4eca-a19f-493ed9705514" containerName="revoke-keys"
Nov 03 18:13:51.103754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103747    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cace8e1-9202-4eca-a19f-493ed9705514" containerName="revoke-keys"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103763    4493 cpu_manager.go:414] "containerMap: removing container" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103768    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103777    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3f68b88-be7d-42e9-8b64-c94c23c9d67f" containerName="payload-pre-processing"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103782    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f68b88-be7d-42e9-8b64-c94c23c9d67f" containerName="payload-pre-processing"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103790    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" containerName="revoke-keys"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103794    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" containerName="revoke-keys"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103810    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8efb4c96-865e-4dd6-a7c1-816911f15d76" containerName="revoke-keys"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103816    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efb4c96-865e-4dd6-a7c1-816911f15d76" containerName="revoke-keys"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103837    4493 cpu_manager.go:414] "containerMap: removing container" podUID="87362f49-a7bb-45ff-a8d4-2fd9acad9728" containerName="payload-processing"
Nov 03 18:13:51.103883 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103843    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87362f49-a7bb-45ff-a8d4-2fd9acad9728" containerName="payload-processing"
Nov 03 18:13:51.104241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103963    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cace8e1-9202-4eca-a19f-493ed9705514" containerName="revoke-keys"
Nov 03 18:13:51.104241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103973    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="63387678-dbfd-43c5-87a1-ef0b581346af" containerName="main"
Nov 03 18:13:51.104241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103982    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f68b88-be7d-42e9-8b64-c94c23c9d67f" containerName="payload-pre-processing"
Nov 03 18:13:51.104241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103989    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" containerName="revoke-keys"
Nov 03 18:13:51.104241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.103996    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efb4c96-865e-4dd6-a7c1-816911f15d76" containerName="revoke-keys"
Nov 03 18:13:51.104241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.104005    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="87362f49-a7bb-45ff-a8d4-2fd9acad9728" containerName="payload-processing"
Nov 03 18:13:51.112564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.112539    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.166016 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.165986    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"]
Nov 03 18:13:51.213036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.213008    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:13:51.213036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.213021    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:13:51.213036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.213029    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:13:51.218970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.218939    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:13:51.235520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.235487    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpv8j\" (UniqueName: \"kubernetes.io/projected/deb1b7f8-553c-4110-857f-884439149867-kube-api-access-lpv8j\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.235702 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.235683    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb1b7f8-553c-4110-857f-884439149867-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.235860 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.235837    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb1b7f8-553c-4110-857f-884439149867-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.337678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.337639    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpv8j\" (UniqueName: \"kubernetes.io/projected/deb1b7f8-553c-4110-857f-884439149867-kube-api-access-lpv8j\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.337841 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.337765    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb1b7f8-553c-4110-857f-884439149867-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.337933 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.337856    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb1b7f8-553c-4110-857f-884439149867-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.338492 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.338472    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb1b7f8-553c-4110-857f-884439149867-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.340311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.340283    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb1b7f8-553c-4110-857f-884439149867-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.401215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.401151    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpv8j\" (UniqueName: \"kubernetes.io/projected/deb1b7f8-553c-4110-857f-884439149867-kube-api-access-lpv8j\") pod \"maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.428139 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.428107    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:51.814592 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:51.814568    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"]
Nov 03 18:13:51.816903 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:13:51.816841    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb1b7f8_553c_4110_857f_884439149867.slice/crio-e295a6d3c11a7886cebc9db8fbce0bd8ee22728055d6dc78f5cef6aecac069a3 WatchSource:0}: Error finding container e295a6d3c11a7886cebc9db8fbce0bd8ee22728055d6dc78f5cef6aecac069a3: Status 404 returned error can't find the container with id e295a6d3c11a7886cebc9db8fbce0bd8ee22728055d6dc78f5cef6aecac069a3
Nov 03 18:13:52.158768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:52.158670    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj" event={"ID":"deb1b7f8-553c-4110-857f-884439149867","Type":"ContainerStarted","Data":"e295a6d3c11a7886cebc9db8fbce0bd8ee22728055d6dc78f5cef6aecac069a3"}
Nov 03 18:13:53.172462 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:53.172369    4493 generic.go:356] "Generic (PLEG): container finished" podID="deb1b7f8-553c-4110-857f-884439149867" containerID="64daefbe97cfd817c7aa5e418b3041174a1761aa9524cca6ae08410525290671" exitCode=0
Nov 03 18:13:53.172462 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:53.172410    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj" event={"ID":"deb1b7f8-553c-4110-857f-884439149867","Type":"ContainerDied","Data":"64daefbe97cfd817c7aa5e418b3041174a1761aa9524cca6ae08410525290671"}
Nov 03 18:13:54.501538 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.501496    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:13:54.505554 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.505516    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb1b7f8-553c-4110-857f-884439149867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb1b7f8-553c-4110-857f-884439149867-maas-api-xks-ca\") pod \"deb1b7f8-553c-4110-857f-884439149867\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") "
Nov 03 18:13:54.505702 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.505683    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb1b7f8-553c-4110-857f-884439149867-kube-api-access-lpv8j\" (UniqueName: \"kubernetes.io/projected/deb1b7f8-553c-4110-857f-884439149867-kube-api-access-lpv8j\") pod \"deb1b7f8-553c-4110-857f-884439149867\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") "
Nov 03 18:13:54.505766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.505721    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb1b7f8-553c-4110-857f-884439149867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb1b7f8-553c-4110-857f-884439149867-maas-api-openshift-ca\") pod \"deb1b7f8-553c-4110-857f-884439149867\" (UID: \"deb1b7f8-553c-4110-857f-884439149867\") "
Nov 03 18:13:54.506495 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.506463    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb1b7f8-553c-4110-857f-884439149867-maas-api-openshift-ca" pod "deb1b7f8-553c-4110-857f-884439149867" (UID: "deb1b7f8-553c-4110-857f-884439149867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:13:54.506886 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.506845    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deb1b7f8-553c-4110-857f-884439149867-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:54.508370 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.508328    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb1b7f8-553c-4110-857f-884439149867-maas-api-xks-ca" pod "deb1b7f8-553c-4110-857f-884439149867" (UID: "deb1b7f8-553c-4110-857f-884439149867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:13:54.509347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.509311    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb1b7f8-553c-4110-857f-884439149867-kube-api-access-lpv8j" pod "deb1b7f8-553c-4110-857f-884439149867" (UID: "deb1b7f8-553c-4110-857f-884439149867"). InnerVolumeSpecName "kube-api-access-lpv8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:13:54.610896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.610837    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpv8j\" (UniqueName: \"kubernetes.io/projected/deb1b7f8-553c-4110-857f-884439149867-kube-api-access-lpv8j\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:54.611196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:54.611166    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deb1b7f8-553c-4110-857f-884439149867-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:13:55.331965 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:55.331924    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj" event={"ID":"deb1b7f8-553c-4110-857f-884439149867","Type":"ContainerDied","Data":"e295a6d3c11a7886cebc9db8fbce0bd8ee22728055d6dc78f5cef6aecac069a3"}
Nov 03 18:13:55.332257 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:55.332237    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e295a6d3c11a7886cebc9db8fbce0bd8ee22728055d6dc78f5cef6aecac069a3"
Nov 03 18:13:55.332439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:13:55.332426    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"
Nov 03 18:14:01.315108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:01.315074    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"]
Nov 03 18:14:01.315597 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:01.315320    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" containerID="cri-o://45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a" gracePeriod=30
Nov 03 18:14:02.046298 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:02.046245    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"]
Nov 03 18:14:05.491219 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:05.491177    4493 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:14:05.491624 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:05.491252    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:14:06.579188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.579129    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 18:14:06.622657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622620    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-socket\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.622657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622663    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-kube-api-access-jrxfx\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-kube-api-access-jrxfx\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.622950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622696    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istiod-ca-cert\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.622950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622721    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-credential-socket\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.622950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622742    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-certs\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.622950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622763    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-podinfo\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.622950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622908    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-token\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-token\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.622950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.622947    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-data\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.623272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623029    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-socket" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.623272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623125    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-envoy\") pod \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\" (UID: \"1f7c15d7-8dcb-4644-9c6e-cade90d240e4\") "
Nov 03 18:14:06.623272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623142    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-certs" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.623272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623157    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-credential-socket" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.623272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623222    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istiod-ca-cert" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:06.623533 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623363    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-data" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.623862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623813    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.623862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623839    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.623862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623854    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.623862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623887    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.624165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.623901    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.625327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.625301    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-podinfo" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:06.625445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.625338    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-envoy" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:06.625445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.625367    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-kube-api-access-jrxfx" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "kube-api-access-jrxfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:06.625445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.625423    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-token" pod "1f7c15d7-8dcb-4644-9c6e-cade90d240e4" (UID: "1f7c15d7-8dcb-4644-9c6e-cade90d240e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:06.725299 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.725272    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.725299 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.725299    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxfx\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-kube-api-access-jrxfx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.725474 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.725314    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.725474 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:06.725328    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f7c15d7-8dcb-4644-9c6e-cade90d240e4-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:06.991851 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:14:06.991814    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:14:07.046680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.046633    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm" podUID="13316d7a-0f35-41af-ae5c-52febd2e3132" containerName="payload-processing" containerID="cri-o://633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9" gracePeriod=25
Nov 03 18:14:07.304125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.304096    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:14:07.331695 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.331657    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13316d7a-0f35-41af-ae5c-52febd2e3132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13316d7a-0f35-41af-ae5c-52febd2e3132-plugins-config-volume\") pod \"13316d7a-0f35-41af-ae5c-52febd2e3132\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") "
Nov 03 18:14:07.331918 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.331720    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13316d7a-0f35-41af-ae5c-52febd2e3132-kube-api-access-w9nfw\" (UniqueName: \"kubernetes.io/projected/13316d7a-0f35-41af-ae5c-52febd2e3132-kube-api-access-w9nfw\") pod \"13316d7a-0f35-41af-ae5c-52febd2e3132\" (UID: \"13316d7a-0f35-41af-ae5c-52febd2e3132\") "
Nov 03 18:14:07.332142 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.332117    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13316d7a-0f35-41af-ae5c-52febd2e3132-plugins-config-volume" pod "13316d7a-0f35-41af-ae5c-52febd2e3132" (UID: "13316d7a-0f35-41af-ae5c-52febd2e3132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:07.334166 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.334139    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13316d7a-0f35-41af-ae5c-52febd2e3132-kube-api-access-w9nfw" pod "13316d7a-0f35-41af-ae5c-52febd2e3132" (UID: "13316d7a-0f35-41af-ae5c-52febd2e3132"). InnerVolumeSpecName "kube-api-access-w9nfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:07.409926 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.409866    4493 generic.go:356] "Generic (PLEG): container finished" podID="13316d7a-0f35-41af-ae5c-52febd2e3132" containerID="633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9" exitCode=0
Nov 03 18:14:07.410125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.409938    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"
Nov 03 18:14:07.410125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.409971    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm" event={"ID":"13316d7a-0f35-41af-ae5c-52febd2e3132","Type":"ContainerDied","Data":"633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9"}
Nov 03 18:14:07.410125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.410060    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm" event={"ID":"13316d7a-0f35-41af-ae5c-52febd2e3132","Type":"ContainerDied","Data":"a37cd187d6ccca168b7d0e09b665cc2aaf449ca2b4263f4757b9dc5b52cea618"}
Nov 03 18:14:07.410125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.410074    4493 scope.go:122] "RemoveContainer" containerID="633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9"
Nov 03 18:14:07.411805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.411733    4493 generic.go:356] "Generic (PLEG): container finished" podID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerID="45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a" exitCode=0
Nov 03 18:14:07.411805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.411800    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" event={"ID":"1f7c15d7-8dcb-4644-9c6e-cade90d240e4","Type":"ContainerDied","Data":"45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a"}
Nov 03 18:14:07.412051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.411840    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"
Nov 03 18:14:07.412051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.411849    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs" event={"ID":"1f7c15d7-8dcb-4644-9c6e-cade90d240e4","Type":"ContainerDied","Data":"92896bf1687f27d7922ebf994b03af9817e03d3e00aff082d57cb0155caa8220"}
Nov 03 18:14:07.421675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.421659    4493 scope.go:122] "RemoveContainer" containerID="633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9"
Nov 03 18:14:07.421981 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:14:07.421962    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9\": container with ID starting with 633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9 not found: ID does not exist" containerID="633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9"
Nov 03 18:14:07.422027 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.421991    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9"} err="failed to get container status \"633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9\": rpc error: code = NotFound desc = could not find container \"633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9\": container with ID starting with 633212e693cdc42e9675ad8e9a7a03c0b1f47079f4bd43143e136432d9690ad9 not found: ID does not exist"
Nov 03 18:14:07.422027 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.422009    4493 scope.go:122] "RemoveContainer" containerID="45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a"
Nov 03 18:14:07.433524 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.433460    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13316d7a-0f35-41af-ae5c-52febd2e3132-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:07.433524 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.433498    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nfw\" (UniqueName: \"kubernetes.io/projected/13316d7a-0f35-41af-ae5c-52febd2e3132-kube-api-access-w9nfw\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:07.435348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.434980    4493 scope.go:122] "RemoveContainer" containerID="45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a"
Nov 03 18:14:07.435541 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:14:07.435484    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a\": container with ID starting with 45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a not found: ID does not exist" containerID="45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a"
Nov 03 18:14:07.435803 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.435554    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a"} err="failed to get container status \"45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a\": rpc error: code = NotFound desc = could not find container \"45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a\": container with ID starting with 45ddde74a4ac19e2d5bfe2111acba485932151167f939cc6937558240276562a not found: ID does not exist"
Nov 03 18:14:07.545080 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.545041    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"]
Nov 03 18:14:07.550573 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.550503    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-hlvcs"]
Nov 03 18:14:07.663513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.663479    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"]
Nov 03 18:14:07.670789 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:07.670755    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-640dc9-d4f98757-ppznm"]
Nov 03 18:14:08.713330 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:08.713296    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13316d7a-0f35-41af-ae5c-52febd2e3132" path="/var/lib/kubelet/pods/13316d7a-0f35-41af-ae5c-52febd2e3132/volumes"
Nov 03 18:14:08.714005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:08.713986    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" path="/var/lib/kubelet/pods/1f7c15d7-8dcb-4644-9c6e-cade90d240e4/volumes"
Nov 03 18:14:16.176449 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:16.176410    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"]
Nov 03 18:14:16.226472 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:16.226436    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ed3e95c-qzplh"]
Nov 03 18:14:16.713031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:16.713001    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac" path="/var/lib/kubelet/pods/ae8d69ff-4876-4f5d-8f2a-4a9daf4080ac/volumes"
Nov 03 18:14:28.825189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:28.825149    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"]
Nov 03 18:14:28.875297 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:28.875266    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-07f297-547rv"]
Nov 03 18:14:30.712430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:30.712399    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e620dc5-37e8-4dbc-823b-9d4b967bbb6c" path="/var/lib/kubelet/pods/3e620dc5-37e8-4dbc-823b-9d4b967bbb6c/volumes"
Nov 03 18:14:36.993914 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:14:36.993838    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:14:39.127183 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:39.127147    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"]
Nov 03 18:14:39.127673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:39.127394    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy" containerID="cri-o://38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6" gracePeriod=30
Nov 03 18:14:41.167073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:41.167035    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"]
Nov 03 18:14:41.217134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:41.217094    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-640dc9-bxzhj"]
Nov 03 18:14:42.713310 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:42.713283    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb1b7f8-553c-4110-857f-884439149867" path="/var/lib/kubelet/pods/deb1b7f8-553c-4110-857f-884439149867/volumes"
Nov 03 18:14:44.402334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.402270    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 18:14:44.581935 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.581865    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-kube-api-access-sxvqf\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-kube-api-access-sxvqf\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.581965    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-socket\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582031    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-certs\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582090    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a008924-c63c-4bea-9dcf-438e683276be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a008924-c63c-4bea-9dcf-438e683276be-istio-podinfo\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582286 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582154    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-data\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582343 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582297    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-istio-token\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-istio-token\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582346    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-socket" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.582404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582367    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-certs" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.582512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582377    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-envoy\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582565 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582508    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-credential-socket\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.582716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582684    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-data" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.583020 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.582995    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-credential-socket" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.583708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.583448    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a008924-c63c-4bea-9dcf-438e683276be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a008924-c63c-4bea-9dcf-438e683276be-istiod-ca-cert\") pod \"8a008924-c63c-4bea-9dcf-438e683276be\" (UID: \"8a008924-c63c-4bea-9dcf-438e683276be\") "
Nov 03 18:14:44.584437 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.584203    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.584437 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.584227    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.584437 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.584241    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.584437 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.584257    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.585054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.585030    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-kube-api-access-sxvqf" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "kube-api-access-sxvqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:44.585054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.585042    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a008924-c63c-4bea-9dcf-438e683276be-istiod-ca-cert" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:44.585201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.585183    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-istio-token" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:44.585248 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.585228    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-envoy" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:44.586295 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.586266    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a008924-c63c-4bea-9dcf-438e683276be-istio-podinfo" pod "8a008924-c63c-4bea-9dcf-438e683276be" (UID: "8a008924-c63c-4bea-9dcf-438e683276be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:44.668702 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.668619    4493 generic.go:356] "Generic (PLEG): container finished" podID="8a008924-c63c-4bea-9dcf-438e683276be" containerID="38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6" exitCode=0
Nov 03 18:14:44.668702 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.668674    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"
Nov 03 18:14:44.668901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.668698    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" event={"ID":"8a008924-c63c-4bea-9dcf-438e683276be","Type":"ContainerDied","Data":"38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6"}
Nov 03 18:14:44.668901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.668759    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l" event={"ID":"8a008924-c63c-4bea-9dcf-438e683276be","Type":"ContainerDied","Data":"5ccf3dd8f15066076e95ab9bb3a937f2dfc1faedee9279c5ddbb90b3b1aa5964"}
Nov 03 18:14:44.668901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.668783    4493 scope.go:122] "RemoveContainer" containerID="38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6"
Nov 03 18:14:44.679946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.679926    4493 scope.go:122] "RemoveContainer" containerID="38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6"
Nov 03 18:14:44.680257 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:14:44.680237    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6\": container with ID starting with 38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6 not found: ID does not exist" containerID="38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6"
Nov 03 18:14:44.680320 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.680266    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6"} err="failed to get container status \"38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6\": rpc error: code = NotFound desc = could not find container \"38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6\": container with ID starting with 38be41f0f9410459800682ffb53ebb6a522a701496f27bf3637b4028d9d502b6 not found: ID does not exist"
Nov 03 18:14:44.685336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.685302    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a008924-c63c-4bea-9dcf-438e683276be-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.685336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.685339    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.685480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.685349    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a008924-c63c-4bea-9dcf-438e683276be-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.685480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.685360    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a008924-c63c-4bea-9dcf-438e683276be-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.685480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.685368    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxvqf\" (UniqueName: \"kubernetes.io/projected/8a008924-c63c-4bea-9dcf-438e683276be-kube-api-access-sxvqf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:44.843579 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.843536    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"]
Nov 03 18:14:44.846600 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:44.846574    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-f589l"]
Nov 03 18:14:46.712637 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:46.712596    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a008924-c63c-4bea-9dcf-438e683276be" path="/var/lib/kubelet/pods/8a008924-c63c-4bea-9dcf-438e683276be/volumes"
Nov 03 18:14:49.963913 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:49.963862    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"]
Nov 03 18:14:49.964530 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:49.964112    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy" containerID="cri-o://123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae" gracePeriod=30
Nov 03 18:14:51.824061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.823980    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"]
Nov 03 18:14:51.824665 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824643    4493 cpu_manager.go:414] "containerMap: removing container" podUID="13316d7a-0f35-41af-ae5c-52febd2e3132" containerName="payload-processing"
Nov 03 18:14:51.824772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824667    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13316d7a-0f35-41af-ae5c-52febd2e3132" containerName="payload-processing"
Nov 03 18:14:51.824772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824689    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy"
Nov 03 18:14:51.824772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824697    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy"
Nov 03 18:14:51.824772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824725    4493 cpu_manager.go:414] "containerMap: removing container" podUID="deb1b7f8-553c-4110-857f-884439149867" containerName="revoke-keys"
Nov 03 18:14:51.824772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824733    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb1b7f8-553c-4110-857f-884439149867" containerName="revoke-keys"
Nov 03 18:14:51.824772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824772    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy"
Nov 03 18:14:51.825102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824779    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy"
Nov 03 18:14:51.825102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824965    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a008924-c63c-4bea-9dcf-438e683276be" containerName="istio-proxy"
Nov 03 18:14:51.825102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824985    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="13316d7a-0f35-41af-ae5c-52febd2e3132" containerName="payload-processing"
Nov 03 18:14:51.825102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.824997    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7c15d7-8dcb-4644-9c6e-cade90d240e4" containerName="istio-proxy"
Nov 03 18:14:51.825102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.825009    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb1b7f8-553c-4110-857f-884439149867" containerName="revoke-keys"
Nov 03 18:14:51.829827 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.829803    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.875771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.875731    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.876002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.875789    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.876128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.876104    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qbqv\" (UniqueName: \"kubernetes.io/projected/ed91a619-a867-4b69-8073-9b89ef8d5561-kube-api-access-6qbqv\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.887830 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.887797    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"]
Nov 03 18:14:51.935105 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.935068    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:14:51.935105 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.935068    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:14:51.935105 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.935103    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:14:51.935361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.935196    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:14:51.977676 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.977631    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.977922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.977714    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.978022 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.977957    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qbqv\" (UniqueName: \"kubernetes.io/projected/ed91a619-a867-4b69-8073-9b89ef8d5561-kube-api-access-6qbqv\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.978440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.978406    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:51.980277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:51.980253    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:52.043782 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:52.043743    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qbqv\" (UniqueName: \"kubernetes.io/projected/ed91a619-a867-4b69-8073-9b89ef8d5561-kube-api-access-6qbqv\") pod \"maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:52.142795 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:52.142715    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:52.529765 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:52.529729    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"]
Nov 03 18:14:52.533058 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:14:52.533025    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded91a619_a867_4b69_8073_9b89ef8d5561.slice/crio-3948881ac3ef771c870958d9daef2e5aeb9ad8032079f1b142680286af9e7552 WatchSource:0}: Error finding container 3948881ac3ef771c870958d9daef2e5aeb9ad8032079f1b142680286af9e7552: Status 404 returned error can't find the container with id 3948881ac3ef771c870958d9daef2e5aeb9ad8032079f1b142680286af9e7552
Nov 03 18:14:52.724837 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:52.724804    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk" event={"ID":"ed91a619-a867-4b69-8073-9b89ef8d5561","Type":"ContainerStarted","Data":"3948881ac3ef771c870958d9daef2e5aeb9ad8032079f1b142680286af9e7552"}
Nov 03 18:14:53.732741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:53.732704    4493 generic.go:356] "Generic (PLEG): container finished" podID="ed91a619-a867-4b69-8073-9b89ef8d5561" containerID="3be6609e031fe702e481570e9172b9e0d5f75c759ca996e7313ba009c34e96b8" exitCode=0
Nov 03 18:14:53.733217 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:53.732778    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk" event={"ID":"ed91a619-a867-4b69-8073-9b89ef8d5561","Type":"ContainerDied","Data":"3be6609e031fe702e481570e9172b9e0d5f75c759ca996e7313ba009c34e96b8"}
Nov 03 18:14:54.894648 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:54.894618    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:55.014372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.014305    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed91a619-a867-4b69-8073-9b89ef8d5561-kube-api-access-6qbqv\" (UniqueName: \"kubernetes.io/projected/ed91a619-a867-4b69-8073-9b89ef8d5561-kube-api-access-6qbqv\") pod \"ed91a619-a867-4b69-8073-9b89ef8d5561\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") "
Nov 03 18:14:55.014498 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.014467    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-xks-ca\") pod \"ed91a619-a867-4b69-8073-9b89ef8d5561\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") "
Nov 03 18:14:55.014566 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.014555    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-openshift-ca\") pod \"ed91a619-a867-4b69-8073-9b89ef8d5561\" (UID: \"ed91a619-a867-4b69-8073-9b89ef8d5561\") "
Nov 03 18:14:55.014965 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.014941    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-openshift-ca" pod "ed91a619-a867-4b69-8073-9b89ef8d5561" (UID: "ed91a619-a867-4b69-8073-9b89ef8d5561"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:55.016586 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.016563    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-xks-ca" pod "ed91a619-a867-4b69-8073-9b89ef8d5561" (UID: "ed91a619-a867-4b69-8073-9b89ef8d5561"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:14:55.016679 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.016596    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed91a619-a867-4b69-8073-9b89ef8d5561-kube-api-access-6qbqv" pod "ed91a619-a867-4b69-8073-9b89ef8d5561" (UID: "ed91a619-a867-4b69-8073-9b89ef8d5561"). InnerVolumeSpecName "kube-api-access-6qbqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:55.116091 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.116044    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.116091 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.116091    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qbqv\" (UniqueName: \"kubernetes.io/projected/ed91a619-a867-4b69-8073-9b89ef8d5561-kube-api-access-6qbqv\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.116332 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.116107    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed91a619-a867-4b69-8073-9b89ef8d5561-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.227244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.227221    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:14:55.318739 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.318650    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-token\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.318961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.318772    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-data\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.318961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.318793    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-podinfo\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.318961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.318907    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-envoy\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.319148 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.318988    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6cd61f-0b32-4936-9c60-158663ebe336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cd61f-0b32-4936-9c60-158663ebe336-istiod-ca-cert\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.319148 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319081    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-credential-socket\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.319148 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319133    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-certs\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.319314 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319161    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-socket\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.319314 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319163    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-data" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:55.319314 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319233    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-kube-api-access-rjddd\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-kube-api-access-rjddd\") pod \"2b6cd61f-0b32-4936-9c60-158663ebe336\" (UID: \"2b6cd61f-0b32-4936-9c60-158663ebe336\") "
Nov 03 18:14:55.319473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319389    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-credential-socket" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:55.319473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319427    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6cd61f-0b32-4936-9c60-158663ebe336-istiod-ca-cert" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:14:55.319473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319412    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-certs" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:55.319618 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.319471    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-socket" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:55.320228 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.320114    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cd61f-0b32-4936-9c60-158663ebe336-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.320228 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.320143    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.320228 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.320163    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.320228 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.320176    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.320228 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.320190    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.321404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.321378    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-token" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:55.321404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.321373    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-envoy" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:14:55.321607 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.321559    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-podinfo" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:14:55.321755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.321734    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-kube-api-access-rjddd" pod "2b6cd61f-0b32-4936-9c60-158663ebe336" (UID: "2b6cd61f-0b32-4936-9c60-158663ebe336"). InnerVolumeSpecName "kube-api-access-rjddd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:14:55.421796 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.421756    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjddd\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-kube-api-access-rjddd\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.421796 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.421790    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.421796 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.421800    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.422071 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.421808    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cd61f-0b32-4936-9c60-158663ebe336-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:14:55.511060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.511022    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"]
Nov 03 18:14:55.527152 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.527119    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"]
Nov 03 18:14:55.746413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.746376    4493 generic.go:356] "Generic (PLEG): container finished" podID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerID="123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae" exitCode=0
Nov 03 18:14:55.746614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.746422    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"
Nov 03 18:14:55.746614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.746458    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" event={"ID":"2b6cd61f-0b32-4936-9c60-158663ebe336","Type":"ContainerDied","Data":"123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae"}
Nov 03 18:14:55.746614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.746514    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk" event={"ID":"2b6cd61f-0b32-4936-9c60-158663ebe336","Type":"ContainerDied","Data":"c3f496d8a00fa0b706e41399bb4950ac2f6d4c83fa012f44a88bbd3538a015a8"}
Nov 03 18:14:55.746614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.746525    4493 scope.go:122] "RemoveContainer" containerID="123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae"
Nov 03 18:14:55.748009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.747994    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"
Nov 03 18:14:55.748121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.748015    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk" event={"ID":"ed91a619-a867-4b69-8073-9b89ef8d5561","Type":"ContainerDied","Data":"3948881ac3ef771c870958d9daef2e5aeb9ad8032079f1b142680286af9e7552"}
Nov 03 18:14:55.748121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.748048    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3948881ac3ef771c870958d9daef2e5aeb9ad8032079f1b142680286af9e7552"
Nov 03 18:14:55.759306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.759283    4493 scope.go:122] "RemoveContainer" containerID="067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658"
Nov 03 18:14:55.768494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.768474    4493 scope.go:122] "RemoveContainer" containerID="123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae"
Nov 03 18:14:55.768781 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:14:55.768763    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae\": container with ID starting with 123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae not found: ID does not exist" containerID="123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae"
Nov 03 18:14:55.768843 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.768789    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae"} err="failed to get container status \"123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae\": rpc error: code = NotFound desc = could not find container \"123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae\": container with ID starting with 123bec0575dfb8e64667cf3e2b7d1dcbb6a45f1263b010023537a07748ffe9ae not found: ID does not exist"
Nov 03 18:14:55.768843 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.768806    4493 scope.go:122] "RemoveContainer" containerID="067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658"
Nov 03 18:14:55.769114 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:14:55.769096    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658\": container with ID starting with 067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658 not found: ID does not exist" containerID="067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658"
Nov 03 18:14:55.769167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.769119    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658"} err="failed to get container status \"067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658\": rpc error: code = NotFound desc = could not find container \"067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658\": container with ID starting with 067071de7d605865d159b60eee453d249d2aed53d19e764b4d86830bd4eb8658 not found: ID does not exist"
Nov 03 18:14:55.978978 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.978933    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"]
Nov 03 18:14:55.993554 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:55.993512    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-640dc9-openshift-default-5bc4ddbfc9-vgxfk"]
Nov 03 18:14:56.713638 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:56.713606    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" path="/var/lib/kubelet/pods/2b6cd61f-0b32-4936-9c60-158663ebe336/volumes"
Nov 03 18:14:57.577513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.577477    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"]
Nov 03 18:14:57.577987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.577973    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ed91a619-a867-4b69-8073-9b89ef8d5561" containerName="revoke-keys"
Nov 03 18:14:57.578047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.577988    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed91a619-a867-4b69-8073-9b89ef8d5561" containerName="revoke-keys"
Nov 03 18:14:57.578047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.578003    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy"
Nov 03 18:14:57.578047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.578008    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy"
Nov 03 18:14:57.578047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.578020    4493 cpu_manager.go:414] "containerMap: removing container" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy"
Nov 03 18:14:57.578047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.578025    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy"
Nov 03 18:14:57.578203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.578120    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy"
Nov 03 18:14:57.578203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.578134    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed91a619-a867-4b69-8073-9b89ef8d5561" containerName="revoke-keys"
Nov 03 18:14:57.578203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.578140    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6cd61f-0b32-4936-9c60-158663ebe336" containerName="istio-proxy"
Nov 03 18:14:57.582680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.582663    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.641616 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.641541    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"]
Nov 03 18:14:57.649370 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.649332    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.649588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.649569    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbvsl\" (UniqueName: \"kubernetes.io/projected/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-kube-api-access-fbvsl\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.649680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.649599    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.688290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.688247    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:14:57.688426 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.688305    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:14:57.688426 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.688313    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:14:57.688426 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.688393    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:14:57.751248 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.751213    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbvsl\" (UniqueName: \"kubernetes.io/projected/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-kube-api-access-fbvsl\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.751424 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.751261    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.751424 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.751374    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.752004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.751978    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.753836 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.753817    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.815700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.815662    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbvsl\" (UniqueName: \"kubernetes.io/projected/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-kube-api-access-fbvsl\") pod \"maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:57.895180 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:57.895097    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:14:58.273527 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:58.273502    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"]
Nov 03 18:14:58.274583 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:14:58.274560    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c5412f_fb67_47ac_91b5_bff1a6561a2c.slice/crio-c3edfc3b9ccc8a3b26ec616709053e0a17fda94b55e970116ac82687ee862fa6 WatchSource:0}: Error finding container c3edfc3b9ccc8a3b26ec616709053e0a17fda94b55e970116ac82687ee862fa6: Status 404 returned error can't find the container with id c3edfc3b9ccc8a3b26ec616709053e0a17fda94b55e970116ac82687ee862fa6
Nov 03 18:14:58.771715 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:58.771684    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl" event={"ID":"d7c5412f-fb67-47ac-91b5-bff1a6561a2c","Type":"ContainerStarted","Data":"3c858a814a99383f959dcd4c1f28f2b9c9b3295413f8c3c267e1daadd1950151"}
Nov 03 18:14:58.772121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:58.771744    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl" event={"ID":"d7c5412f-fb67-47ac-91b5-bff1a6561a2c","Type":"ContainerStarted","Data":"c3edfc3b9ccc8a3b26ec616709053e0a17fda94b55e970116ac82687ee862fa6"}
Nov 03 18:14:58.893684 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:58.893585    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl" podStartSLOduration=1.893570876 podStartE2EDuration="1.893570876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:58.892406841 +0000 UTC m=+2478.351577827" watchObservedRunningTime="2026-11-03 18:14:58.893570876 +0000 UTC m=+2478.352741861"
Nov 03 18:14:59.781136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:59.781100    4493 generic.go:356] "Generic (PLEG): container finished" podID="d7c5412f-fb67-47ac-91b5-bff1a6561a2c" containerID="3c858a814a99383f959dcd4c1f28f2b9c9b3295413f8c3c267e1daadd1950151" exitCode=0
Nov 03 18:14:59.781573 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:14:59.781190    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl" event={"ID":"d7c5412f-fb67-47ac-91b5-bff1a6561a2c","Type":"ContainerDied","Data":"3c858a814a99383f959dcd4c1f28f2b9c9b3295413f8c3c267e1daadd1950151"}
Nov 03 18:15:00.171489 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.171409    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"]
Nov 03 18:15:00.177386 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.177217    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"]
Nov 03 18:15:00.177540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.177486    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.186479 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.186448    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"]
Nov 03 18:15:00.188339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.188301    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.191687 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.191666    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"]
Nov 03 18:15:00.191808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.191789    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.197153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.197135    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"]
Nov 03 18:15:00.197260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.197244    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.200707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.200689    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.232809 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.232789    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"]
Nov 03 18:15:00.235324 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.235303    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"]
Nov 03 18:15:00.276533 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.276503    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"]
Nov 03 18:15:00.279947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.279919    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"]
Nov 03 18:15:00.286312 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286287    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4081a-942d-4c14-95ee-39a7ada193b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.286443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286433    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvmj\" (UniqueName: \"kubernetes.io/projected/6bd4081a-942d-4c14-95ee-39a7ada193b1-kube-api-access-zpvmj\") pod \"maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.286492 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286458    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt9gc\" (UniqueName: \"kubernetes.io/projected/e432deb9-c69b-4766-b635-b6b6baee7b1a-kube-api-access-kt9gc\") pod \"maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.286567 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286543    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49352363-7b25-4daa-8986-744f4e269cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.286623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286592    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2ml\" (UniqueName: \"kubernetes.io/projected/6784968f-f0cc-4bc2-a870-4e42548bf837-kube-api-access-xn2ml\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.286710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286690    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a92c0a-f5cd-4d86-9544-4129e2860646-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.286773 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286759    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6784968f-f0cc-4bc2-a870-4e42548bf837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.286811 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286789    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gsh\" (UniqueName: \"kubernetes.io/projected/49352363-7b25-4daa-8986-744f4e269cf5-kube-api-access-f4gsh\") pod \"maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.286854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286815    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e432deb9-c69b-4766-b635-b6b6baee7b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.286925 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.286861    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fkb\" (UniqueName: \"kubernetes.io/projected/99a92c0a-f5cd-4d86-9544-4129e2860646-kube-api-access-62fkb\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.327322 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.327290    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"]
Nov 03 18:15:00.388480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388434    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvmj\" (UniqueName: \"kubernetes.io/projected/6bd4081a-942d-4c14-95ee-39a7ada193b1-kube-api-access-zpvmj\") pod \"maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.388480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388487    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt9gc\" (UniqueName: \"kubernetes.io/projected/e432deb9-c69b-4766-b635-b6b6baee7b1a-kube-api-access-kt9gc\") pod \"maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.388741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388512    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49352363-7b25-4daa-8986-744f4e269cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.388741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388529    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2ml\" (UniqueName: \"kubernetes.io/projected/6784968f-f0cc-4bc2-a870-4e42548bf837-kube-api-access-xn2ml\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.388741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388550    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a92c0a-f5cd-4d86-9544-4129e2860646-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.388741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388576    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6784968f-f0cc-4bc2-a870-4e42548bf837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.388741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388612    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gsh\" (UniqueName: \"kubernetes.io/projected/49352363-7b25-4daa-8986-744f4e269cf5-kube-api-access-f4gsh\") pod \"maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.389066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388753    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e432deb9-c69b-4766-b635-b6b6baee7b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.389066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388788    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fkb\" (UniqueName: \"kubernetes.io/projected/99a92c0a-f5cd-4d86-9544-4129e2860646-kube-api-access-62fkb\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.389066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.388894    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4081a-942d-4c14-95ee-39a7ada193b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.389481 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.389456    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a92c0a-f5cd-4d86-9544-4129e2860646-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.389532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.389473    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49352363-7b25-4daa-8986-744f4e269cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.389532 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.389456    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6784968f-f0cc-4bc2-a870-4e42548bf837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.389602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.389529    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e432deb9-c69b-4766-b635-b6b6baee7b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.389602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.389547    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4081a-942d-4c14-95ee-39a7ada193b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.452349 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.452272    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gsh\" (UniqueName: \"kubernetes.io/projected/49352363-7b25-4daa-8986-744f4e269cf5-kube-api-access-f4gsh\") pod \"maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.452509 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.452346    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2ml\" (UniqueName: \"kubernetes.io/projected/6784968f-f0cc-4bc2-a870-4e42548bf837-kube-api-access-xn2ml\") pod \"maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.452509 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.452364    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fkb\" (UniqueName: \"kubernetes.io/projected/99a92c0a-f5cd-4d86-9544-4129e2860646-kube-api-access-62fkb\") pod \"maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.452629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.452535    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt9gc\" (UniqueName: \"kubernetes.io/projected/e432deb9-c69b-4766-b635-b6b6baee7b1a-kube-api-access-kt9gc\") pod \"maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.452629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.452578    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvmj\" (UniqueName: \"kubernetes.io/projected/6bd4081a-942d-4c14-95ee-39a7ada193b1-kube-api-access-zpvmj\") pod \"maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.499665 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.499640    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:00.506818 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.506797    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:00.516377 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.516355    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:00.524405 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.523046    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:00.530210 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.530032    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:00.931811 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.931784    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:15:00.945229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.945208    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"]
Nov 03 18:15:00.946220 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:00.946196    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd4081a_942d_4c14_95ee_39a7ada193b1.slice/crio-adddad74ca9e2514f8fd9b57a00dbf9a53e5790c045727ef6d54b243d08f741f WatchSource:0}: Error finding container adddad74ca9e2514f8fd9b57a00dbf9a53e5790c045727ef6d54b243d08f741f: Status 404 returned error can't find the container with id adddad74ca9e2514f8fd9b57a00dbf9a53e5790c045727ef6d54b243d08f741f
Nov 03 18:15:00.957807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.956671    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"]
Nov 03 18:15:00.959145 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:00.959108    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a92c0a_f5cd_4d86_9544_4129e2860646.slice/crio-d2e0d6a4fccd526ee082f62d71d9f4efaf66b1918103f3236fdd87d2ef312a19 WatchSource:0}: Error finding container d2e0d6a4fccd526ee082f62d71d9f4efaf66b1918103f3236fdd87d2ef312a19: Status 404 returned error can't find the container with id d2e0d6a4fccd526ee082f62d71d9f4efaf66b1918103f3236fdd87d2ef312a19
Nov 03 18:15:00.959268 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.959171    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"]
Nov 03 18:15:00.960246 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:00.960222    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode432deb9_c69b_4766_b635_b6b6baee7b1a.slice/crio-be036f87880a11aae32cdb1bfe11d759ffca04995035d4a5623fc5399ed7b1fe WatchSource:0}: Error finding container be036f87880a11aae32cdb1bfe11d759ffca04995035d4a5623fc5399ed7b1fe: Status 404 returned error can't find the container with id be036f87880a11aae32cdb1bfe11d759ffca04995035d4a5623fc5399ed7b1fe
Nov 03 18:15:00.963100 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.963075    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"]
Nov 03 18:15:00.965592 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:00.965567    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49352363_7b25_4daa_8986_744f4e269cf5.slice/crio-d4094a87e86cf90d7fd1ae5f0ad374d6f4178c2416566bdeda1152d40ac492f0 WatchSource:0}: Error finding container d4094a87e86cf90d7fd1ae5f0ad374d6f4178c2416566bdeda1152d40ac492f0: Status 404 returned error can't find the container with id d4094a87e86cf90d7fd1ae5f0ad374d6f4178c2416566bdeda1152d40ac492f0
Nov 03 18:15:00.998576 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.998551    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-openshift-ca\") pod \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") "
Nov 03 18:15:00.998755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.998738    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-kube-api-access-fbvsl\" (UniqueName: \"kubernetes.io/projected/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-kube-api-access-fbvsl\") pod \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") "
Nov 03 18:15:00.998859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.998779    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-xks-ca\") pod \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\" (UID: \"d7c5412f-fb67-47ac-91b5-bff1a6561a2c\") "
Nov 03 18:15:00.999070 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.999040    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-openshift-ca" pod "d7c5412f-fb67-47ac-91b5-bff1a6561a2c" (UID: "d7c5412f-fb67-47ac-91b5-bff1a6561a2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:00.999255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:00.999237    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.000865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.000843    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-kube-api-access-fbvsl" pod "d7c5412f-fb67-47ac-91b5-bff1a6561a2c" (UID: "d7c5412f-fb67-47ac-91b5-bff1a6561a2c"). InnerVolumeSpecName "kube-api-access-fbvsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:01.000865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.000886    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-xks-ca" pod "d7c5412f-fb67-47ac-91b5-bff1a6561a2c" (UID: "d7c5412f-fb67-47ac-91b5-bff1a6561a2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:01.005490 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.005471    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"]
Nov 03 18:15:01.006583 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:01.006564    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6784968f_f0cc_4bc2_a870_4e42548bf837.slice/crio-8cdf49cc5de9f56f6c86b9551a05e6b1acee17051d4ed3fd0bfab0901c72cf5d WatchSource:0}: Error finding container 8cdf49cc5de9f56f6c86b9551a05e6b1acee17051d4ed3fd0bfab0901c72cf5d: Status 404 returned error can't find the container with id 8cdf49cc5de9f56f6c86b9551a05e6b1acee17051d4ed3fd0bfab0901c72cf5d
Nov 03 18:15:01.100504 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.100404    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.100504 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.100443    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbvsl\" (UniqueName: \"kubernetes.io/projected/d7c5412f-fb67-47ac-91b5-bff1a6561a2c-kube-api-access-fbvsl\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:01.797045 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.797015    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl" event={"ID":"d7c5412f-fb67-47ac-91b5-bff1a6561a2c","Type":"ContainerDied","Data":"c3edfc3b9ccc8a3b26ec616709053e0a17fda94b55e970116ac82687ee862fa6"}
Nov 03 18:15:01.797249 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.797069    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3edfc3b9ccc8a3b26ec616709053e0a17fda94b55e970116ac82687ee862fa6"
Nov 03 18:15:01.797249 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.797025    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"
Nov 03 18:15:01.798692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.798665    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" event={"ID":"6bd4081a-942d-4c14-95ee-39a7ada193b1","Type":"ContainerStarted","Data":"6cf82e8cd80c955d96584da276a72ee8cd4808d27c5f8a523ac734a43b540e70"}
Nov 03 18:15:01.798818 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.798709    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" event={"ID":"6bd4081a-942d-4c14-95ee-39a7ada193b1","Type":"ContainerStarted","Data":"adddad74ca9e2514f8fd9b57a00dbf9a53e5790c045727ef6d54b243d08f741f"}
Nov 03 18:15:01.800238 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.800217    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" event={"ID":"99a92c0a-f5cd-4d86-9544-4129e2860646","Type":"ContainerStarted","Data":"dc5075fd10dd6498906dc2feefd9fd5910ee9a58f4a678524c31cff237b1bc84"}
Nov 03 18:15:01.800333 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.800256    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" event={"ID":"99a92c0a-f5cd-4d86-9544-4129e2860646","Type":"ContainerStarted","Data":"d2e0d6a4fccd526ee082f62d71d9f4efaf66b1918103f3236fdd87d2ef312a19"}
Nov 03 18:15:01.801921 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.801863    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" event={"ID":"6784968f-f0cc-4bc2-a870-4e42548bf837","Type":"ContainerStarted","Data":"5c1bd0ce6f339636f006e46aa9212dc81a598c25f8e989f335a695485e350aed"}
Nov 03 18:15:01.802020 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.801943    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" event={"ID":"6784968f-f0cc-4bc2-a870-4e42548bf837","Type":"ContainerStarted","Data":"8cdf49cc5de9f56f6c86b9551a05e6b1acee17051d4ed3fd0bfab0901c72cf5d"}
Nov 03 18:15:01.803670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.803641    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" event={"ID":"e432deb9-c69b-4766-b635-b6b6baee7b1a","Type":"ContainerStarted","Data":"5076543d9a03bbcd9107e9384a093b0d51cf4c09679b535fbf49ab8475fe6815"}
Nov 03 18:15:01.803774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.803688    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" event={"ID":"e432deb9-c69b-4766-b635-b6b6baee7b1a","Type":"ContainerStarted","Data":"be036f87880a11aae32cdb1bfe11d759ffca04995035d4a5623fc5399ed7b1fe"}
Nov 03 18:15:01.805642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.805622    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" event={"ID":"49352363-7b25-4daa-8986-744f4e269cf5","Type":"ContainerStarted","Data":"3a3f311150522132a21c3b4a988d1b94830ca5c7f309b208e7f08e4a4b56d4f6"}
Nov 03 18:15:01.805785 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.805665    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" event={"ID":"49352363-7b25-4daa-8986-744f4e269cf5","Type":"ContainerStarted","Data":"d4094a87e86cf90d7fd1ae5f0ad374d6f4178c2416566bdeda1152d40ac492f0"}
Nov 03 18:15:01.920103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:01.920037    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" podStartSLOduration=1.920020036 podStartE2EDuration="1.920020036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:01.917454006 +0000 UTC m=+2481.376624996" watchObservedRunningTime="2026-11-03 18:15:01.920020036 +0000 UTC m=+2481.379191023"
Nov 03 18:15:02.011294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.011255    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"]
Nov 03 18:15:02.025102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.025070    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"]
Nov 03 18:15:02.042210 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.042154    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" podStartSLOduration=2.042105908 podStartE2EDuration="2.042105908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.036398913 +0000 UTC m=+2481.495569899" watchObservedRunningTime="2026-11-03 18:15:02.042105908 +0000 UTC m=+2481.501276895"
Nov 03 18:15:02.188905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.180920    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" podStartSLOduration=2.180896621 podStartE2EDuration="2.180896621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.158727455 +0000 UTC m=+2481.617898441" watchObservedRunningTime="2026-11-03 18:15:02.180896621 +0000 UTC m=+2481.640067607"
Nov 03 18:15:02.280685 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.280628    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" podStartSLOduration=2.280608806 podStartE2EDuration="2.280608806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.279267396 +0000 UTC m=+2481.738438381" watchObservedRunningTime="2026-11-03 18:15:02.280608806 +0000 UTC m=+2481.739779791"
Nov 03 18:15:02.398591 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.398492    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" podStartSLOduration=2.398476709 podStartE2EDuration="2.398476709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.397740627 +0000 UTC m=+2481.856911615" watchObservedRunningTime="2026-11-03 18:15:02.398476709 +0000 UTC m=+2481.857647695"
Nov 03 18:15:02.813966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.813925    4493 generic.go:356] "Generic (PLEG): container finished" podID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerID="6cf82e8cd80c955d96584da276a72ee8cd4808d27c5f8a523ac734a43b540e70" exitCode=60
Nov 03 18:15:02.814166 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.814014    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" event={"ID":"6bd4081a-942d-4c14-95ee-39a7ada193b1","Type":"ContainerDied","Data":"6cf82e8cd80c955d96584da276a72ee8cd4808d27c5f8a523ac734a43b540e70"}
Nov 03 18:15:02.814332 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.814315    4493 scope.go:122] "RemoveContainer" containerID="6cf82e8cd80c955d96584da276a72ee8cd4808d27c5f8a523ac734a43b540e70"
Nov 03 18:15:02.815797 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.815774    4493 generic.go:356] "Generic (PLEG): container finished" podID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerID="dc5075fd10dd6498906dc2feefd9fd5910ee9a58f4a678524c31cff237b1bc84" exitCode=60
Nov 03 18:15:02.815942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.815857    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" event={"ID":"99a92c0a-f5cd-4d86-9544-4129e2860646","Type":"ContainerDied","Data":"dc5075fd10dd6498906dc2feefd9fd5910ee9a58f4a678524c31cff237b1bc84"}
Nov 03 18:15:02.816125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.816111    4493 scope.go:122] "RemoveContainer" containerID="dc5075fd10dd6498906dc2feefd9fd5910ee9a58f4a678524c31cff237b1bc84"
Nov 03 18:15:02.817680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.817644    4493 generic.go:356] "Generic (PLEG): container finished" podID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerID="5c1bd0ce6f339636f006e46aa9212dc81a598c25f8e989f335a695485e350aed" exitCode=60
Nov 03 18:15:02.817969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.817942    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" event={"ID":"6784968f-f0cc-4bc2-a870-4e42548bf837","Type":"ContainerDied","Data":"5c1bd0ce6f339636f006e46aa9212dc81a598c25f8e989f335a695485e350aed"}
Nov 03 18:15:02.818074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.818040    4493 scope.go:122] "RemoveContainer" containerID="5c1bd0ce6f339636f006e46aa9212dc81a598c25f8e989f335a695485e350aed"
Nov 03 18:15:02.819823 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.819802    4493 generic.go:356] "Generic (PLEG): container finished" podID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerID="5076543d9a03bbcd9107e9384a093b0d51cf4c09679b535fbf49ab8475fe6815" exitCode=60
Nov 03 18:15:02.819928 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.819893    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" event={"ID":"e432deb9-c69b-4766-b635-b6b6baee7b1a","Type":"ContainerDied","Data":"5076543d9a03bbcd9107e9384a093b0d51cf4c09679b535fbf49ab8475fe6815"}
Nov 03 18:15:02.820081 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.820054    4493 scope.go:122] "RemoveContainer" containerID="5076543d9a03bbcd9107e9384a093b0d51cf4c09679b535fbf49ab8475fe6815"
Nov 03 18:15:02.821746 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.821724    4493 generic.go:356] "Generic (PLEG): container finished" podID="49352363-7b25-4daa-8986-744f4e269cf5" containerID="3a3f311150522132a21c3b4a988d1b94830ca5c7f309b208e7f08e4a4b56d4f6" exitCode=60
Nov 03 18:15:02.821849 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.821807    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" event={"ID":"49352363-7b25-4daa-8986-744f4e269cf5","Type":"ContainerDied","Data":"3a3f311150522132a21c3b4a988d1b94830ca5c7f309b208e7f08e4a4b56d4f6"}
Nov 03 18:15:02.822001 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:02.821986    4493 scope.go:122] "RemoveContainer" containerID="3a3f311150522132a21c3b4a988d1b94830ca5c7f309b208e7f08e4a4b56d4f6"
Nov 03 18:15:03.832166 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.832131    4493 generic.go:356] "Generic (PLEG): container finished" podID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerID="860ceef9b0adbc8861490cc21c5a2cd765980500bbbb1c16f6eab70d44d85b2f" exitCode=60
Nov 03 18:15:03.832528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.832205    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" event={"ID":"e432deb9-c69b-4766-b635-b6b6baee7b1a","Type":"ContainerDied","Data":"860ceef9b0adbc8861490cc21c5a2cd765980500bbbb1c16f6eab70d44d85b2f"}
Nov 03 18:15:03.832528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.832260    4493 scope.go:122] "RemoveContainer" containerID="5076543d9a03bbcd9107e9384a093b0d51cf4c09679b535fbf49ab8475fe6815"
Nov 03 18:15:03.832656 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.832534    4493 scope.go:122] "RemoveContainer" containerID="860ceef9b0adbc8861490cc21c5a2cd765980500bbbb1c16f6eab70d44d85b2f"
Nov 03 18:15:03.832925 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:03.832899    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74_odh-ai-gateway-infra(e432deb9-c69b-4766-b635-b6b6baee7b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a"
Nov 03 18:15:03.834209 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.834188    4493 generic.go:356] "Generic (PLEG): container finished" podID="49352363-7b25-4daa-8986-744f4e269cf5" containerID="2410595ceee9b941931de584af952cbda81ff68ef2a189e5c49233d75253d8f1" exitCode=60
Nov 03 18:15:03.834292 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.834270    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" event={"ID":"49352363-7b25-4daa-8986-744f4e269cf5","Type":"ContainerDied","Data":"2410595ceee9b941931de584af952cbda81ff68ef2a189e5c49233d75253d8f1"}
Nov 03 18:15:03.834548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.834534    4493 scope.go:122] "RemoveContainer" containerID="2410595ceee9b941931de584af952cbda81ff68ef2a189e5c49233d75253d8f1"
Nov 03 18:15:03.834861 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:03.834838    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8_odh-ai-gateway-infra(49352363-7b25-4daa-8986-744f4e269cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" podUID="49352363-7b25-4daa-8986-744f4e269cf5"
Nov 03 18:15:03.836308 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.836273    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" event={"ID":"6bd4081a-942d-4c14-95ee-39a7ada193b1","Type":"ContainerStarted","Data":"c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317"}
Nov 03 18:15:03.836528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.836502    4493 scope.go:122] "RemoveContainer" containerID="c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317"
Nov 03 18:15:03.836848 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:03.836796    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h_odh-ai-gateway-infra(6bd4081a-942d-4c14-95ee-39a7ada193b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1"
Nov 03 18:15:03.838189 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.838168    4493 generic.go:356] "Generic (PLEG): container finished" podID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerID="9a97d90a92c2846b629899be755c2871641b5f41b786e7b13d666da87629cef7" exitCode=60
Nov 03 18:15:03.838294 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.838216    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" event={"ID":"99a92c0a-f5cd-4d86-9544-4129e2860646","Type":"ContainerDied","Data":"9a97d90a92c2846b629899be755c2871641b5f41b786e7b13d666da87629cef7"}
Nov 03 18:15:03.838458 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.838442    4493 scope.go:122] "RemoveContainer" containerID="9a97d90a92c2846b629899be755c2871641b5f41b786e7b13d666da87629cef7"
Nov 03 18:15:03.838685 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:03.838666    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n_odh-ai-gateway-infra(99a92c0a-f5cd-4d86-9544-4129e2860646)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646"
Nov 03 18:15:03.840253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:03.840230    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" event={"ID":"6784968f-f0cc-4bc2-a870-4e42548bf837","Type":"ContainerStarted","Data":"3f1f5eba32223f58dc5cacc2e48e8f68ce2d47eae272f8fdf3ca1dbc42362711"}
Nov 03 18:15:04.014074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.014043    4493 scope.go:122] "RemoveContainer" containerID="3a3f311150522132a21c3b4a988d1b94830ca5c7f309b208e7f08e4a4b56d4f6"
Nov 03 18:15:04.027635 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.027614    4493 scope.go:122] "RemoveContainer" containerID="dc5075fd10dd6498906dc2feefd9fd5910ee9a58f4a678524c31cff237b1bc84"
Nov 03 18:15:04.716054 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.716022    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"]
Nov 03 18:15:04.852801 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.852772    4493 scope.go:122] "RemoveContainer" containerID="860ceef9b0adbc8861490cc21c5a2cd765980500bbbb1c16f6eab70d44d85b2f"
Nov 03 18:15:04.853532 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:04.853502    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74_odh-ai-gateway-infra(e432deb9-c69b-4766-b635-b6b6baee7b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a"
Nov 03 18:15:04.854429 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.854408    4493 scope.go:122] "RemoveContainer" containerID="2410595ceee9b941931de584af952cbda81ff68ef2a189e5c49233d75253d8f1"
Nov 03 18:15:04.854653 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:04.854632    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8_odh-ai-gateway-infra(49352363-7b25-4daa-8986-744f4e269cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" podUID="49352363-7b25-4daa-8986-744f4e269cf5"
Nov 03 18:15:04.856169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.856142    4493 generic.go:356] "Generic (PLEG): container finished" podID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerID="c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317" exitCode=60
Nov 03 18:15:04.856265 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.856232    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" event={"ID":"6bd4081a-942d-4c14-95ee-39a7ada193b1","Type":"ContainerDied","Data":"c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317"}
Nov 03 18:15:04.856331 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.856322    4493 scope.go:122] "RemoveContainer" containerID="6cf82e8cd80c955d96584da276a72ee8cd4808d27c5f8a523ac734a43b540e70"
Nov 03 18:15:04.856466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.856446    4493 scope.go:122] "RemoveContainer" containerID="c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317"
Nov 03 18:15:04.856808 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:04.856787    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h_odh-ai-gateway-infra(6bd4081a-942d-4c14-95ee-39a7ada193b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1"
Nov 03 18:15:04.858284 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.858258    4493 scope.go:122] "RemoveContainer" containerID="9a97d90a92c2846b629899be755c2871641b5f41b786e7b13d666da87629cef7"
Nov 03 18:15:04.858453 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:04.858435    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n_odh-ai-gateway-infra(99a92c0a-f5cd-4d86-9544-4129e2860646)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646"
Nov 03 18:15:04.859905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.859854    4493 generic.go:356] "Generic (PLEG): container finished" podID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerID="3f1f5eba32223f58dc5cacc2e48e8f68ce2d47eae272f8fdf3ca1dbc42362711" exitCode=60
Nov 03 18:15:04.859992 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.859912    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" event={"ID":"6784968f-f0cc-4bc2-a870-4e42548bf837","Type":"ContainerDied","Data":"3f1f5eba32223f58dc5cacc2e48e8f68ce2d47eae272f8fdf3ca1dbc42362711"}
Nov 03 18:15:04.860131 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.860116    4493 scope.go:122] "RemoveContainer" containerID="3f1f5eba32223f58dc5cacc2e48e8f68ce2d47eae272f8fdf3ca1dbc42362711"
Nov 03 18:15:04.860323 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:04.860307    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r_odh-ai-gateway-infra(6784968f-f0cc-4bc2-a870-4e42548bf837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837"
Nov 03 18:15:04.869801 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:04.869779    4493 scope.go:122] "RemoveContainer" containerID="5c1bd0ce6f339636f006e46aa9212dc81a598c25f8e989f335a695485e350aed"
Nov 03 18:15:05.868591 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:05.868561    4493 scope.go:122] "RemoveContainer" containerID="c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317"
Nov 03 18:15:05.869057 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:05.868915    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h_odh-ai-gateway-infra(6bd4081a-942d-4c14-95ee-39a7ada193b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1"
Nov 03 18:15:05.870191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:05.870169    4493 scope.go:122] "RemoveContainer" containerID="3f1f5eba32223f58dc5cacc2e48e8f68ce2d47eae272f8fdf3ca1dbc42362711"
Nov 03 18:15:05.870433 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:05.870416    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r_odh-ai-gateway-infra(6784968f-f0cc-4bc2-a870-4e42548bf837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837"
Nov 03 18:15:07.085259 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:07.085220    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:15:09.030620 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:09.030578    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"]
Nov 03 18:15:09.031100 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:09.030803    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy" containerID="cri-o://683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1" gracePeriod=30
Nov 03 18:15:09.051557 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:09.051515    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"]
Nov 03 18:15:09.051743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:09.051721    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy" containerID="cri-o://18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10" gracePeriod=30
Nov 03 18:15:09.717266 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:09.717220    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" podUID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" containerName="payload-processing" containerID="cri-o://3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa" gracePeriod=25
Nov 03 18:15:10.532370 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.532320    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main" containerID="cri-o://828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e" gracePeriod=45
Nov 03 18:15:10.547193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.547161    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main" containerID="cri-o://14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233" gracePeriod=45
Nov 03 18:15:10.854158 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.854134    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 18:15:10.857837 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.857811    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 18:15:10.900485 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900451    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-dshm\") pod \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") "
Nov 03 18:15:10.900652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900496    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-model-cache\") pod \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") "
Nov 03 18:15:10.900652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900545    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tmp-dir\") pod \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") "
Nov 03 18:15:10.900730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900657    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-kube-api-access-c5jjn\" (UniqueName: \"kubernetes.io/projected/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-kube-api-access-c5jjn\") pod \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") "
Nov 03 18:15:10.900730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900722    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-home\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-home\") pod \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") "
Nov 03 18:15:10.900814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900751    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-dshm\") pod \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") "
Nov 03 18:15:10.900814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900786    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tls-certs\" (UniqueName: \"kubernetes.io/secret/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tls-certs\") pod \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") "
Nov 03 18:15:10.900966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900795    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-model-cache" pod "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" (UID: "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.901138 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900826    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tmp-dir\") pod \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") "
Nov 03 18:15:10.901220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.900940    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tmp-dir" pod "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" (UID: "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.901277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901053    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-home" pod "809fc6c4-b67d-4d4e-9225-39b4a2a6757d" (UID: "809fc6c4-b67d-4d4e-9225-39b4a2a6757d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.901277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901068    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tmp-dir" pod "809fc6c4-b67d-4d4e-9225-39b4a2a6757d" (UID: "809fc6c4-b67d-4d4e-9225-39b4a2a6757d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.901277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901205    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-home\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-home\") pod \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") "
Nov 03 18:15:10.901513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901296    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tls-certs\") pod \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\" (UID: \"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae\") "
Nov 03 18:15:10.901513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901399    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-kube-api-access-nfvbv\" (UniqueName: \"kubernetes.io/projected/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-kube-api-access-nfvbv\") pod \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") "
Nov 03 18:15:10.901513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901502    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-model-cache\") pod \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\" (UID: \"809fc6c4-b67d-4d4e-9225-39b4a2a6757d\") "
Nov 03 18:15:10.901757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901548    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-home" pod "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" (UID: "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.902042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.901987    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-model-cache" pod "809fc6c4-b67d-4d4e-9225-39b4a2a6757d" (UID: "809fc6c4-b67d-4d4e-9225-39b4a2a6757d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.902361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.902339    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.902430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.902365    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.902430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.902380    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.902430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.902393    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.902430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.902405    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.902430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.902417    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:10.903488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.903450    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-kube-api-access-c5jjn" pod "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" (UID: "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae"). InnerVolumeSpecName "kube-api-access-c5jjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:10.903581 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.903519    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-dshm" pod "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" (UID: "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.903640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.903602    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-dshm" pod "809fc6c4-b67d-4d4e-9225-39b4a2a6757d" (UID: "809fc6c4-b67d-4d4e-9225-39b4a2a6757d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:10.903832 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.903812    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tls-certs" pod "809fc6c4-b67d-4d4e-9225-39b4a2a6757d" (UID: "809fc6c4-b67d-4d4e-9225-39b4a2a6757d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:10.903922 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.903860    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-kube-api-access-nfvbv" pod "809fc6c4-b67d-4d4e-9225-39b4a2a6757d" (UID: "809fc6c4-b67d-4d4e-9225-39b4a2a6757d"). InnerVolumeSpecName "kube-api-access-nfvbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:10.904260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.904240    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tls-certs" pod "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" (UID: "503a8b5b-5bf6-4f06-8268-3cda7c14b9ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:10.919864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.919835    4493 generic.go:356] "Generic (PLEG): container finished" podID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerID="14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233" exitCode=0
Nov 03 18:15:10.919996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.919898    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"
Nov 03 18:15:10.919996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.919911    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" event={"ID":"809fc6c4-b67d-4d4e-9225-39b4a2a6757d","Type":"ContainerDied","Data":"14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233"}
Nov 03 18:15:10.919996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.919955    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq" event={"ID":"809fc6c4-b67d-4d4e-9225-39b4a2a6757d","Type":"ContainerDied","Data":"d2819368c39aa837266bd850d12ee23a59e52408aa8ff439936ff45e68beb703"}
Nov 03 18:15:10.919996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.919972    4493 scope.go:122] "RemoveContainer" containerID="14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233"
Nov 03 18:15:10.921738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.921634    4493 generic.go:356] "Generic (PLEG): container finished" podID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerID="828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e" exitCode=0
Nov 03 18:15:10.921738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.921677    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"
Nov 03 18:15:10.921911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.921678    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" event={"ID":"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae","Type":"ContainerDied","Data":"828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e"}
Nov 03 18:15:10.921911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.921785    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4" event={"ID":"503a8b5b-5bf6-4f06-8268-3cda7c14b9ae","Type":"ContainerDied","Data":"873cb354d6224402dc213e0a7bc96129641557dba7b8d3f9d69f75227d0b8ff9"}
Nov 03 18:15:10.930544 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.930526    4493 scope.go:122] "RemoveContainer" containerID="14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233"
Nov 03 18:15:10.930864 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:10.930841    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233\": container with ID starting with 14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233 not found: ID does not exist" containerID="14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233"
Nov 03 18:15:10.931002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.930906    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233"} err="failed to get container status \"14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233\": rpc error: code = NotFound desc = could not find container \"14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233\": container with ID starting with 14714fc5a8a2f2c18b386e5fe0233ed0e1e872854572540e611ef965e3e5d233 not found: ID does not exist"
Nov 03 18:15:10.931002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.930929    4493 scope.go:122] "RemoveContainer" containerID="828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e"
Nov 03 18:15:10.940557 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.940536    4493 scope.go:122] "RemoveContainer" containerID="828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e"
Nov 03 18:15:10.940848 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:10.940827    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e\": container with ID starting with 828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e not found: ID does not exist" containerID="828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e"
Nov 03 18:15:10.940966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:10.940855    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e"} err="failed to get container status \"828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e\": rpc error: code = NotFound desc = could not find container \"828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e\": container with ID starting with 828862624bc723e54fd5fb7ab6c744f0cce07e17429c1c1a48fb51ec89cff66e not found: ID does not exist"
Nov 03 18:15:11.003944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.003897    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.003944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.003943    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvbv\" (UniqueName: \"kubernetes.io/projected/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-kube-api-access-nfvbv\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.003944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.003955    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.003944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.003964    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5jjn\" (UniqueName: \"kubernetes.io/projected/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae-kube-api-access-c5jjn\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.004307 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.003972    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.004307 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.003982    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809fc6c4-b67d-4d4e-9225-39b4a2a6757d-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:11.095999 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.095906    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"]
Nov 03 18:15:11.100564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.100538    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/distinct-2-w5-548ca1-kserve-5f85557885-8d5h4"]
Nov 03 18:15:11.285952 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.285916    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"]
Nov 03 18:15:11.291768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.291738    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/premium-w5-548ca1-kserve-b7666bf99-j79kq"]
Nov 03 18:15:11.504712 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.504671    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:15:11.504911 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.504737    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:15:11.719073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.719038    4493 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:15:11.719466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:11.719090    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:15:12.712156 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:12.712125    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" path="/var/lib/kubelet/pods/503a8b5b-5bf6-4f06-8268-3cda7c14b9ae/volumes"
Nov 03 18:15:12.712638 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:12.712623    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" path="/var/lib/kubelet/pods/809fc6c4-b67d-4d4e-9225-39b4a2a6757d/volumes"
Nov 03 18:15:13.387076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:13.387047    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"]
Nov 03 18:15:14.327574 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.327547    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:15:14.358779 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.358752    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:15:14.452461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452422    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-socket\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452490    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-data\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452604    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-kube-api-access-6nhmf\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-kube-api-access-6nhmf\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452632    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-credential-socket\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452661    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-credential-socket\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452735    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-podinfo\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452776    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-envoy\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452825    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f9f592-5d8b-4cd3-9883-aef79b463eca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f9f592-5d8b-4cd3-9883-aef79b463eca-istiod-ca-cert\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452844    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-certs\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452909    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-envoy\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452937    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07be7d18-9942-4d43-889a-0cfe69b59bb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be7d18-9942-4d43-889a-0cfe69b59bb1-istiod-ca-cert\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.452987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.452978    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-token\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-token\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.453828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.453002    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-certs\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.453828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.453049    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-token\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-token\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.453828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.453078    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-data\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.453828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.453100    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-socket\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.453828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.453122    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-kube-api-access-z8dld\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-kube-api-access-z8dld\") pod \"07be7d18-9942-4d43-889a-0cfe69b59bb1\" (UID: \"07be7d18-9942-4d43-889a-0cfe69b59bb1\") "
Nov 03 18:15:14.453828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.453157    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-podinfo\") pod \"57f9f592-5d8b-4cd3-9883-aef79b463eca\" (UID: \"57f9f592-5d8b-4cd3-9883-aef79b463eca\") "
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.459578    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-podinfo" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.459795    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-socket" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.460000    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-data" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.460827    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-data" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.460894    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-socket" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.461394    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07be7d18-9942-4d43-889a-0cfe69b59bb1-istiod-ca-cert" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.461752    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-certs" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.462092    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-credential-socket" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.462233    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-certs" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.462503    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-credential-socket" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.466486    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-kube-api-access-6nhmf" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "kube-api-access-6nhmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.467092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.467033    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f9f592-5d8b-4cd3-9883-aef79b463eca-istiod-ca-cert" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:14.468140 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.468115    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-envoy" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.468262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.468144    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-podinfo" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:15:14.468334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.468252    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-token" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.468334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.468256    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-token" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.468334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.468270    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-kube-api-access-z8dld" pod "07be7d18-9942-4d43-889a-0cfe69b59bb1" (UID: "07be7d18-9942-4d43-889a-0cfe69b59bb1"). InnerVolumeSpecName "kube-api-access-z8dld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:14.469753 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.469729    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-envoy" pod "57f9f592-5d8b-4cd3-9883-aef79b463eca" (UID: "57f9f592-5d8b-4cd3-9883-aef79b463eca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:14.555457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555385    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555414    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555424    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555435    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8dld\" (UniqueName: \"kubernetes.io/projected/07be7d18-9942-4d43-889a-0cfe69b59bb1-kube-api-access-z8dld\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555444    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555453    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555461    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555470    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhmf\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-kube-api-access-6nhmf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555478    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555486    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555494    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555502    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555512    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f9f592-5d8b-4cd3-9883-aef79b463eca-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555520    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555528    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be7d18-9942-4d43-889a-0cfe69b59bb1-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555548    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be7d18-9942-4d43-889a-0cfe69b59bb1-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555556    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f9f592-5d8b-4cd3-9883-aef79b463eca-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.555781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.555565    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f9f592-5d8b-4cd3-9883-aef79b463eca-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:14.706404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.706366    4493 scope.go:122] "RemoveContainer" containerID="2410595ceee9b941931de584af952cbda81ff68ef2a189e5c49233d75253d8f1"
Nov 03 18:15:14.706564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.706532    4493 scope.go:122] "RemoveContainer" containerID="860ceef9b0adbc8861490cc21c5a2cd765980500bbbb1c16f6eab70d44d85b2f"
Nov 03 18:15:14.974100 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.974060    4493 generic.go:356] "Generic (PLEG): container finished" podID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerID="18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10" exitCode=0
Nov 03 18:15:14.974306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.974120    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"
Nov 03 18:15:14.974306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.974153    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" event={"ID":"07be7d18-9942-4d43-889a-0cfe69b59bb1","Type":"ContainerDied","Data":"18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10"}
Nov 03 18:15:14.974306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.974229    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb" event={"ID":"07be7d18-9942-4d43-889a-0cfe69b59bb1","Type":"ContainerDied","Data":"6681e97c305dfad8bd49914a3fd6a9dafb0dcecfdcfe2b549c1a210f07c64c75"}
Nov 03 18:15:14.974306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.974256    4493 scope.go:122] "RemoveContainer" containerID="18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10"
Nov 03 18:15:14.981267 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.981242    4493 generic.go:356] "Generic (PLEG): container finished" podID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerID="683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1" exitCode=0
Nov 03 18:15:14.981405 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.981303    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"
Nov 03 18:15:14.981405 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.981342    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" event={"ID":"57f9f592-5d8b-4cd3-9883-aef79b463eca","Type":"ContainerDied","Data":"683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1"}
Nov 03 18:15:14.981405 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.981385    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z" event={"ID":"57f9f592-5d8b-4cd3-9883-aef79b463eca","Type":"ContainerDied","Data":"2c29e840de0b18b71da18521c7680ef7a86f7d028e7c038f6632b19bae60716d"}
Nov 03 18:15:14.990994 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.990974    4493 scope.go:122] "RemoveContainer" containerID="18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10"
Nov 03 18:15:14.991309 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:14.991289    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10\": container with ID starting with 18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10 not found: ID does not exist" containerID="18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10"
Nov 03 18:15:14.991356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.991318    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10"} err="failed to get container status \"18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10\": rpc error: code = NotFound desc = could not find container \"18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10\": container with ID starting with 18ebc8f8c49cc542ae446eedc8d751be2c3a0672317ac80bf787ff8775404d10 not found: ID does not exist"
Nov 03 18:15:14.991356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:14.991337    4493 scope.go:122] "RemoveContainer" containerID="683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1"
Nov 03 18:15:15.000998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.000965    4493 scope.go:122] "RemoveContainer" containerID="683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1"
Nov 03 18:15:15.001247 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:15.001230    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1\": container with ID starting with 683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1 not found: ID does not exist" containerID="683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1"
Nov 03 18:15:15.001309 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.001255    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1"} err="failed to get container status \"683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1\": rpc error: code = NotFound desc = could not find container \"683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1\": container with ID starting with 683409061d9128c75704aad5fdea571c2197ca9e0757a3bc6f0960242c1c84b1 not found: ID does not exist"
Nov 03 18:15:15.146005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.145972    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"]
Nov 03 18:15:15.152078 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.152050    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-w2sbb"]
Nov 03 18:15:15.267942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.267899    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"]
Nov 03 18:15:15.275915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.275865    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-5np6z"]
Nov 03 18:15:15.992287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.992254    4493 generic.go:356] "Generic (PLEG): container finished" podID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerID="2b9e862dd7070562da3a9becc758cfc533ec82f64d0e626a7653d9566983f33f" exitCode=60
Nov 03 18:15:15.992738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.992328    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" event={"ID":"e432deb9-c69b-4766-b635-b6b6baee7b1a","Type":"ContainerDied","Data":"2b9e862dd7070562da3a9becc758cfc533ec82f64d0e626a7653d9566983f33f"}
Nov 03 18:15:15.992738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.992381    4493 scope.go:122] "RemoveContainer" containerID="860ceef9b0adbc8861490cc21c5a2cd765980500bbbb1c16f6eab70d44d85b2f"
Nov 03 18:15:15.992829 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.992750    4493 scope.go:122] "RemoveContainer" containerID="2b9e862dd7070562da3a9becc758cfc533ec82f64d0e626a7653d9566983f33f"
Nov 03 18:15:15.993112 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:15.993087    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74_odh-ai-gateway-infra(e432deb9-c69b-4766-b635-b6b6baee7b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a"
Nov 03 18:15:15.994409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.994389    4493 generic.go:356] "Generic (PLEG): container finished" podID="49352363-7b25-4daa-8986-744f4e269cf5" containerID="3685f6f6b856559eb898c29f3350f3d88ca2877c9c5fd0b4ec3ba637c34855ce" exitCode=60
Nov 03 18:15:15.994479 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.994454    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" event={"ID":"49352363-7b25-4daa-8986-744f4e269cf5","Type":"ContainerDied","Data":"3685f6f6b856559eb898c29f3350f3d88ca2877c9c5fd0b4ec3ba637c34855ce"}
Nov 03 18:15:15.994815 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:15.994800    4493 scope.go:122] "RemoveContainer" containerID="3685f6f6b856559eb898c29f3350f3d88ca2877c9c5fd0b4ec3ba637c34855ce"
Nov 03 18:15:15.995100 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:15.995082    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8_odh-ai-gateway-infra(49352363-7b25-4daa-8986-744f4e269cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" podUID="49352363-7b25-4daa-8986-744f4e269cf5"
Nov 03 18:15:16.005239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:16.005220    4493 scope.go:122] "RemoveContainer" containerID="2410595ceee9b941931de584af952cbda81ff68ef2a189e5c49233d75253d8f1"
Nov 03 18:15:16.706304 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:16.706278    4493 scope.go:122] "RemoveContainer" containerID="3f1f5eba32223f58dc5cacc2e48e8f68ce2d47eae272f8fdf3ca1dbc42362711"
Nov 03 18:15:16.706494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:16.706395    4493 scope.go:122] "RemoveContainer" containerID="c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317"
Nov 03 18:15:16.714080 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:16.714053    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" path="/var/lib/kubelet/pods/07be7d18-9942-4d43-889a-0cfe69b59bb1/volumes"
Nov 03 18:15:16.714683 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:16.714663    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" path="/var/lib/kubelet/pods/57f9f592-5d8b-4cd3-9883-aef79b463eca/volumes"
Nov 03 18:15:17.036454 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.036422    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" containerID="cri-o://df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048" gracePeriod=45
Nov 03 18:15:17.051405 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.051370    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" containerID="cri-o://e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab" gracePeriod=45
Nov 03 18:15:17.127455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.127430    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"]
Nov 03 18:15:17.129191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.129163    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused"
Nov 03 18:15:17.160862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.160816    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Nov 03 18:15:17.248327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.248298    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"]
Nov 03 18:15:17.306669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.306642    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:17.356801 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.356767    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 18:15:17.401136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.401103    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 18:15:17.401503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.401478    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e432deb9-c69b-4766-b635-b6b6baee7b1a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e432deb9-c69b-4766-b635-b6b6baee7b1a-maas-api-service-ca\") pod \"e432deb9-c69b-4766-b635-b6b6baee7b1a\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") "
Nov 03 18:15:17.401592 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.401531    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e432deb9-c69b-4766-b635-b6b6baee7b1a-kube-api-access-kt9gc\" (UniqueName: \"kubernetes.io/projected/e432deb9-c69b-4766-b635-b6b6baee7b1a-kube-api-access-kt9gc\") pod \"e432deb9-c69b-4766-b635-b6b6baee7b1a\" (UID: \"e432deb9-c69b-4766-b635-b6b6baee7b1a\") "
Nov 03 18:15:17.402432 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.402259    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e432deb9-c69b-4766-b635-b6b6baee7b1a-maas-api-service-ca" pod "e432deb9-c69b-4766-b635-b6b6baee7b1a" (UID: "e432deb9-c69b-4766-b635-b6b6baee7b1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:17.406099 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.405295    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e432deb9-c69b-4766-b635-b6b6baee7b1a-kube-api-access-kt9gc" pod "e432deb9-c69b-4766-b635-b6b6baee7b1a" (UID: "e432deb9-c69b-4766-b635-b6b6baee7b1a"). InnerVolumeSpecName "kube-api-access-kt9gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:17.491009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.490907    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:17.502986 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.502957    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-dshm\") pod \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") "
Nov 03 18:15:17.503180 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503070    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tmp-dir\") pod \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") "
Nov 03 18:15:17.503180 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503092    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tls-certs\" (UniqueName: \"kubernetes.io/secret/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tls-certs\") pod \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") "
Nov 03 18:15:17.503180 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503147    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-home\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-home\") pod \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") "
Nov 03 18:15:17.503362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503233    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-dshm\") pod \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") "
Nov 03 18:15:17.503362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503279    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381f0f27-9512-4d8c-a529-5b8cda13dc7f-kube-api-access-z7ng6\" (UniqueName: \"kubernetes.io/projected/381f0f27-9512-4d8c-a529-5b8cda13dc7f-kube-api-access-z7ng6\") pod \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") "
Nov 03 18:15:17.503362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503321    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-model-cache\") pod \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") "
Nov 03 18:15:17.503362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503347    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tmp-dir" pod "c66cfe66-7c3d-423c-b71e-b28fa4e47d75" (UID: "c66cfe66-7c3d-423c-b71e-b28fa4e47d75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.503362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503352    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-kube-api-access-77kkl\" (UniqueName: \"kubernetes.io/projected/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-kube-api-access-77kkl\") pod \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\" (UID: \"c66cfe66-7c3d-423c-b71e-b28fa4e47d75\") "
Nov 03 18:15:17.503622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503386    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tls-certs\") pod \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") "
Nov 03 18:15:17.503622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503449    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tmp-dir\") pod \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") "
Nov 03 18:15:17.503622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503498    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-home\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-home\") pod \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") "
Nov 03 18:15:17.503622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503510    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-home" pod "c66cfe66-7c3d-423c-b71e-b28fa4e47d75" (UID: "c66cfe66-7c3d-423c-b71e-b28fa4e47d75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.503622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503542    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-model-cache\") pod \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\" (UID: \"381f0f27-9512-4d8c-a529-5b8cda13dc7f\") "
Nov 03 18:15:17.504193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503989    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-model-cache" pod "c66cfe66-7c3d-423c-b71e-b28fa4e47d75" (UID: "c66cfe66-7c3d-423c-b71e-b28fa4e47d75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.504193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.504122    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.504390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.504192    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.504390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.504215    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e432deb9-c69b-4766-b635-b6b6baee7b1a-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.504390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.504213    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-model-cache" pod "381f0f27-9512-4d8c-a529-5b8cda13dc7f" (UID: "381f0f27-9512-4d8c-a529-5b8cda13dc7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.504390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.504231    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt9gc\" (UniqueName: \"kubernetes.io/projected/e432deb9-c69b-4766-b635-b6b6baee7b1a-kube-api-access-kt9gc\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.504390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.504247    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.504390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.503995    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tmp-dir" pod "381f0f27-9512-4d8c-a529-5b8cda13dc7f" (UID: "381f0f27-9512-4d8c-a529-5b8cda13dc7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.504696 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.504564    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-home" pod "381f0f27-9512-4d8c-a529-5b8cda13dc7f" (UID: "381f0f27-9512-4d8c-a529-5b8cda13dc7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.506105 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.506070    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-dshm" pod "381f0f27-9512-4d8c-a529-5b8cda13dc7f" (UID: "381f0f27-9512-4d8c-a529-5b8cda13dc7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.506628 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.506595    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-dshm" pod "c66cfe66-7c3d-423c-b71e-b28fa4e47d75" (UID: "c66cfe66-7c3d-423c-b71e-b28fa4e47d75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:15:17.506720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.506700    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tls-certs" pod "381f0f27-9512-4d8c-a529-5b8cda13dc7f" (UID: "381f0f27-9512-4d8c-a529-5b8cda13dc7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:17.507152 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.507120    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tls-certs" pod "c66cfe66-7c3d-423c-b71e-b28fa4e47d75" (UID: "c66cfe66-7c3d-423c-b71e-b28fa4e47d75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:17.507297 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.507276    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381f0f27-9512-4d8c-a529-5b8cda13dc7f-kube-api-access-z7ng6" pod "381f0f27-9512-4d8c-a529-5b8cda13dc7f" (UID: "381f0f27-9512-4d8c-a529-5b8cda13dc7f"). InnerVolumeSpecName "kube-api-access-z7ng6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:17.508355 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.508331    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-kube-api-access-77kkl" pod "c66cfe66-7c3d-423c-b71e-b28fa4e47d75" (UID: "c66cfe66-7c3d-423c-b71e-b28fa4e47d75"). InnerVolumeSpecName "kube-api-access-77kkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:17.605358 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605270    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49352363-7b25-4daa-8986-744f4e269cf5-kube-api-access-f4gsh\" (UniqueName: \"kubernetes.io/projected/49352363-7b25-4daa-8986-744f4e269cf5-kube-api-access-f4gsh\") pod \"49352363-7b25-4daa-8986-744f4e269cf5\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") "
Nov 03 18:15:17.605548 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605367    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49352363-7b25-4daa-8986-744f4e269cf5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49352363-7b25-4daa-8986-744f4e269cf5-maas-api-service-ca\") pod \"49352363-7b25-4daa-8986-744f4e269cf5\" (UID: \"49352363-7b25-4daa-8986-744f4e269cf5\") "
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605748    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49352363-7b25-4daa-8986-744f4e269cf5-maas-api-service-ca" pod "49352363-7b25-4daa-8986-744f4e269cf5" (UID: "49352363-7b25-4daa-8986-744f4e269cf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605759    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605798    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605812    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605820    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605830    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ng6\" (UniqueName: \"kubernetes.io/projected/381f0f27-9512-4d8c-a529-5b8cda13dc7f-kube-api-access-z7ng6\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605843    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kkl\" (UniqueName: \"kubernetes.io/projected/c66cfe66-7c3d-423c-b71e-b28fa4e47d75-kube-api-access-77kkl\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605855    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605863    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.605896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.605893    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381f0f27-9512-4d8c-a529-5b8cda13dc7f-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.608165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.608131    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49352363-7b25-4daa-8986-744f4e269cf5-kube-api-access-f4gsh" pod "49352363-7b25-4daa-8986-744f4e269cf5" (UID: "49352363-7b25-4daa-8986-744f4e269cf5"). InnerVolumeSpecName "kube-api-access-f4gsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:17.706082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.706048    4493 scope.go:122] "RemoveContainer" containerID="9a97d90a92c2846b629899be755c2871641b5f41b786e7b13d666da87629cef7"
Nov 03 18:15:17.709128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.709094    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49352363-7b25-4daa-8986-744f4e269cf5-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:17.709128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:17.709130    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gsh\" (UniqueName: \"kubernetes.io/projected/49352363-7b25-4daa-8986-744f4e269cf5-kube-api-access-f4gsh\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.013359 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.013326    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"
Nov 03 18:15:18.013359 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.013332    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74" event={"ID":"e432deb9-c69b-4766-b635-b6b6baee7b1a","Type":"ContainerDied","Data":"be036f87880a11aae32cdb1bfe11d759ffca04995035d4a5623fc5399ed7b1fe"}
Nov 03 18:15:18.013616 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.013445    4493 scope.go:122] "RemoveContainer" containerID="2b9e862dd7070562da3a9becc758cfc533ec82f64d0e626a7653d9566983f33f"
Nov 03 18:15:18.014689 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.014671    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"
Nov 03 18:15:18.014822 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.014711    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8" event={"ID":"49352363-7b25-4daa-8986-744f4e269cf5","Type":"ContainerDied","Data":"d4094a87e86cf90d7fd1ae5f0ad374d6f4178c2416566bdeda1152d40ac492f0"}
Nov 03 18:15:18.016295 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.016267    4493 generic.go:356] "Generic (PLEG): container finished" podID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerID="e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab" exitCode=0
Nov 03 18:15:18.016397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.016313    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"
Nov 03 18:15:18.016397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.016358    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" event={"ID":"381f0f27-9512-4d8c-a529-5b8cda13dc7f","Type":"ContainerDied","Data":"e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab"}
Nov 03 18:15:18.016524 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.016402    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8" event={"ID":"381f0f27-9512-4d8c-a529-5b8cda13dc7f","Type":"ContainerDied","Data":"b89bea2dfac4d9a9b018205193b4ccafbf69f9f4555780862cf74d90b8584c13"}
Nov 03 18:15:18.019101 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.019063    4493 generic.go:356] "Generic (PLEG): container finished" podID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerID="50e769eaad0e9eeeee0b19c6cfef458c6f71c0bba90bcec826c96e3225a5eb81" exitCode=60
Nov 03 18:15:18.019203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.019098    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" event={"ID":"6bd4081a-942d-4c14-95ee-39a7ada193b1","Type":"ContainerDied","Data":"50e769eaad0e9eeeee0b19c6cfef458c6f71c0bba90bcec826c96e3225a5eb81"}
Nov 03 18:15:18.019529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.019505    4493 scope.go:122] "RemoveContainer" containerID="50e769eaad0e9eeeee0b19c6cfef458c6f71c0bba90bcec826c96e3225a5eb81"
Nov 03 18:15:18.019913 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:18.019856    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h_odh-ai-gateway-infra(6bd4081a-942d-4c14-95ee-39a7ada193b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1"
Nov 03 18:15:18.023864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.023842    4493 generic.go:356] "Generic (PLEG): container finished" podID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerID="087d4c033c8b2f42a1c9837d42081c57c64c9ec9ae6bc40537afb1dc7d1b12c8" exitCode=60
Nov 03 18:15:18.023998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.023904    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" event={"ID":"6784968f-f0cc-4bc2-a870-4e42548bf837","Type":"ContainerDied","Data":"087d4c033c8b2f42a1c9837d42081c57c64c9ec9ae6bc40537afb1dc7d1b12c8"}
Nov 03 18:15:18.024622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.024300    4493 scope.go:122] "RemoveContainer" containerID="087d4c033c8b2f42a1c9837d42081c57c64c9ec9ae6bc40537afb1dc7d1b12c8"
Nov 03 18:15:18.024622 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:18.024573    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r_odh-ai-gateway-infra(6784968f-f0cc-4bc2-a870-4e42548bf837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837"
Nov 03 18:15:18.025611 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.025590    4493 generic.go:356] "Generic (PLEG): container finished" podID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerID="df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048" exitCode=0
Nov 03 18:15:18.025704 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.025634    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"
Nov 03 18:15:18.025704 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.025678    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" event={"ID":"c66cfe66-7c3d-423c-b71e-b28fa4e47d75","Type":"ContainerDied","Data":"df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048"}
Nov 03 18:15:18.025803 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.025716    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc" event={"ID":"c66cfe66-7c3d-423c-b71e-b28fa4e47d75","Type":"ContainerDied","Data":"1ef783106e2b99f30829e1021e272b0b5c68711fceb1e2c219091142e1b4880e"}
Nov 03 18:15:18.028128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.028107    4493 scope.go:122] "RemoveContainer" containerID="3685f6f6b856559eb898c29f3350f3d88ca2877c9c5fd0b4ec3ba637c34855ce"
Nov 03 18:15:18.046371 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.046352    4493 scope.go:122] "RemoveContainer" containerID="e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab"
Nov 03 18:15:18.056917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.056902    4493 scope.go:122] "RemoveContainer" containerID="e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab"
Nov 03 18:15:18.057203 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:18.057174    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab\": container with ID starting with e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab not found: ID does not exist" containerID="e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab"
Nov 03 18:15:18.057279 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.057204    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab"} err="failed to get container status \"e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab\": rpc error: code = NotFound desc = could not find container \"e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab\": container with ID starting with e3d4a2e8f2d70e96e459f0cd8d8582508d211c2c99699f90c8b16ca531ca3bab not found: ID does not exist"
Nov 03 18:15:18.057279 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.057223    4493 scope.go:122] "RemoveContainer" containerID="c73b0a77df80c47fb2f3cd2814b3df3b4076299231baf06501c5499765232317"
Nov 03 18:15:18.070008 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.069990    4493 scope.go:122] "RemoveContainer" containerID="3f1f5eba32223f58dc5cacc2e48e8f68ce2d47eae272f8fdf3ca1dbc42362711"
Nov 03 18:15:18.082549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.082532    4493 scope.go:122] "RemoveContainer" containerID="df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048"
Nov 03 18:15:18.090817 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.090798    4493 scope.go:122] "RemoveContainer" containerID="df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048"
Nov 03 18:15:18.091151 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:18.091131    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048\": container with ID starting with df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048 not found: ID does not exist" containerID="df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048"
Nov 03 18:15:18.091224 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.091160    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048"} err="failed to get container status \"df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048\": rpc error: code = NotFound desc = could not find container \"df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048\": container with ID starting with df6fdbb42ab2a27b0dc01b5cd0252551aef93fe8f1f0026b6ecc37c09d46f048 not found: ID does not exist"
Nov 03 18:15:18.183655 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.183615    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"]
Nov 03 18:15:18.193126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.193094    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-0bd8cf-29895495-w4x74"]
Nov 03 18:15:18.387445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.387338    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf" podUID="56cc8ddf-6721-428b-972c-68a77603096a" containerName="payload-processing" containerID="cri-o://6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2" gracePeriod=25
Nov 03 18:15:18.426823 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.426794    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"]
Nov 03 18:15:18.433048 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.433019    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/simulator-w2-42c162-kserve-7d6dc99555-rhmp8"]
Nov 03 18:15:18.547221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.547187    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"]
Nov 03 18:15:18.549866 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.549833    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/premium-w2-42c162-kserve-6b8dcd7854-lh5qc"]
Nov 03 18:15:18.653989 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.653933    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 18:15:18.713622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.713588    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" path="/var/lib/kubelet/pods/381f0f27-9512-4d8c-a529-5b8cda13dc7f/volumes"
Nov 03 18:15:18.714193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.714173    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" path="/var/lib/kubelet/pods/c66cfe66-7c3d-423c-b71e-b28fa4e47d75/volumes"
Nov 03 18:15:18.714534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.714521    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" path="/var/lib/kubelet/pods/e432deb9-c69b-4766-b635-b6b6baee7b1a/volumes"
Nov 03 18:15:18.746198 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.746166    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56cc8ddf-6721-428b-972c-68a77603096a-kube-api-access-c5rqt\" (UniqueName: \"kubernetes.io/projected/56cc8ddf-6721-428b-972c-68a77603096a-kube-api-access-c5rqt\") pod \"56cc8ddf-6721-428b-972c-68a77603096a\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") "
Nov 03 18:15:18.746399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.746382    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56cc8ddf-6721-428b-972c-68a77603096a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cc8ddf-6721-428b-972c-68a77603096a-plugins-config-volume\") pod \"56cc8ddf-6721-428b-972c-68a77603096a\" (UID: \"56cc8ddf-6721-428b-972c-68a77603096a\") "
Nov 03 18:15:18.746735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.746714    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56cc8ddf-6721-428b-972c-68a77603096a-plugins-config-volume" pod "56cc8ddf-6721-428b-972c-68a77603096a" (UID: "56cc8ddf-6721-428b-972c-68a77603096a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:18.748240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.748224    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56cc8ddf-6721-428b-972c-68a77603096a-kube-api-access-c5rqt" pod "56cc8ddf-6721-428b-972c-68a77603096a" (UID: "56cc8ddf-6721-428b-972c-68a77603096a"). InnerVolumeSpecName "kube-api-access-c5rqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:18.781978 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.781941    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"]
Nov 03 18:15:18.793021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.792990    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ca6b33-29895495-vtdn8"]
Nov 03 18:15:18.824835 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.824797    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" podUID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:18.847726 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.847695    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56cc8ddf-6721-428b-972c-68a77603096a-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:18.847726 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:18.847723    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rqt\" (UniqueName: \"kubernetes.io/projected/56cc8ddf-6721-428b-972c-68a77603096a-kube-api-access-c5rqt\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.047387 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.047350    4493 generic.go:356] "Generic (PLEG): container finished" podID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerID="a28a6316c8a65f1e105ff8466dee61c74b12727fb90bb9e1aa4c2ce573a91518" exitCode=60
Nov 03 18:15:19.047862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.047460    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" event={"ID":"99a92c0a-f5cd-4d86-9544-4129e2860646","Type":"ContainerDied","Data":"a28a6316c8a65f1e105ff8466dee61c74b12727fb90bb9e1aa4c2ce573a91518"}
Nov 03 18:15:19.047862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.047521    4493 scope.go:122] "RemoveContainer" containerID="9a97d90a92c2846b629899be755c2871641b5f41b786e7b13d666da87629cef7"
Nov 03 18:15:19.048049 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.048031    4493 scope.go:122] "RemoveContainer" containerID="a28a6316c8a65f1e105ff8466dee61c74b12727fb90bb9e1aa4c2ce573a91518"
Nov 03 18:15:19.048427 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:19.048403    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n_odh-ai-gateway-infra(99a92c0a-f5cd-4d86-9544-4129e2860646)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646"
Nov 03 18:15:19.051981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.051955    4493 generic.go:356] "Generic (PLEG): container finished" podID="56cc8ddf-6721-428b-972c-68a77603096a" containerID="6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2" exitCode=0
Nov 03 18:15:19.052153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.051990    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"
Nov 03 18:15:19.052153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.052034    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf" event={"ID":"56cc8ddf-6721-428b-972c-68a77603096a","Type":"ContainerDied","Data":"6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2"}
Nov 03 18:15:19.052153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.052100    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf" event={"ID":"56cc8ddf-6721-428b-972c-68a77603096a","Type":"ContainerDied","Data":"dea1e3c135494d88cbed18c1f10b16383c47e349c7236e1fdb96a14e6f7a658b"}
Nov 03 18:15:19.064786 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.064766    4493 scope.go:122] "RemoveContainer" containerID="6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2"
Nov 03 18:15:19.073980 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.073963    4493 scope.go:122] "RemoveContainer" containerID="6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2"
Nov 03 18:15:19.074238 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:19.074219    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2\": container with ID starting with 6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2 not found: ID does not exist" containerID="6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2"
Nov 03 18:15:19.074298 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.074246    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2"} err="failed to get container status \"6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2\": rpc error: code = NotFound desc = could not find container \"6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2\": container with ID starting with 6c9a39b00f6749fd314f4d5785fd077862cb758050e8b569a537d49946e1cdf2 not found: ID does not exist"
Nov 03 18:15:19.285967 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.285933    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"]
Nov 03 18:15:19.292608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.292579    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-42c162-c89b85959-zrfrf"]
Nov 03 18:15:19.321736 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.321670    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"]
Nov 03 18:15:19.471692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.471663    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:19.659388 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.659256    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bd4081a-942d-4c14-95ee-39a7ada193b1-kube-api-access-zpvmj\" (UniqueName: \"kubernetes.io/projected/6bd4081a-942d-4c14-95ee-39a7ada193b1-kube-api-access-zpvmj\") pod \"6bd4081a-942d-4c14-95ee-39a7ada193b1\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") "
Nov 03 18:15:19.659388 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.659358    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bd4081a-942d-4c14-95ee-39a7ada193b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4081a-942d-4c14-95ee-39a7ada193b1-maas-api-service-ca\") pod \"6bd4081a-942d-4c14-95ee-39a7ada193b1\" (UID: \"6bd4081a-942d-4c14-95ee-39a7ada193b1\") "
Nov 03 18:15:19.660205 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.660174    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd4081a-942d-4c14-95ee-39a7ada193b1-maas-api-service-ca" pod "6bd4081a-942d-4c14-95ee-39a7ada193b1" (UID: "6bd4081a-942d-4c14-95ee-39a7ada193b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:19.662151 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.662126    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd4081a-942d-4c14-95ee-39a7ada193b1-kube-api-access-zpvmj" pod "6bd4081a-942d-4c14-95ee-39a7ada193b1" (UID: "6bd4081a-942d-4c14-95ee-39a7ada193b1"). InnerVolumeSpecName "kube-api-access-zpvmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:19.678517 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.678493    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"]
Nov 03 18:15:19.760990 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.760953    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpvmj\" (UniqueName: \"kubernetes.io/projected/6bd4081a-942d-4c14-95ee-39a7ada193b1-kube-api-access-zpvmj\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.760990 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.760992    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd4081a-942d-4c14-95ee-39a7ada193b1-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.816063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.816038    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:19.963021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.962991    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6784968f-f0cc-4bc2-a870-4e42548bf837-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6784968f-f0cc-4bc2-a870-4e42548bf837-maas-api-service-ca\") pod \"6784968f-f0cc-4bc2-a870-4e42548bf837\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") "
Nov 03 18:15:19.963214 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.963194    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6784968f-f0cc-4bc2-a870-4e42548bf837-kube-api-access-xn2ml\" (UniqueName: \"kubernetes.io/projected/6784968f-f0cc-4bc2-a870-4e42548bf837-kube-api-access-xn2ml\") pod \"6784968f-f0cc-4bc2-a870-4e42548bf837\" (UID: \"6784968f-f0cc-4bc2-a870-4e42548bf837\") "
Nov 03 18:15:19.963382 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.963359    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6784968f-f0cc-4bc2-a870-4e42548bf837-maas-api-service-ca" pod "6784968f-f0cc-4bc2-a870-4e42548bf837" (UID: "6784968f-f0cc-4bc2-a870-4e42548bf837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:19.963663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.963631    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6784968f-f0cc-4bc2-a870-4e42548bf837-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:19.965516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:19.965484    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6784968f-f0cc-4bc2-a870-4e42548bf837-kube-api-access-xn2ml" pod "6784968f-f0cc-4bc2-a870-4e42548bf837" (UID: "6784968f-f0cc-4bc2-a870-4e42548bf837"). InnerVolumeSpecName "kube-api-access-xn2ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:20.062343 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.062309    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"
Nov 03 18:15:20.062805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.062336    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h" event={"ID":"6bd4081a-942d-4c14-95ee-39a7ada193b1","Type":"ContainerDied","Data":"adddad74ca9e2514f8fd9b57a00dbf9a53e5790c045727ef6d54b243d08f741f"}
Nov 03 18:15:20.062805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.062403    4493 scope.go:122] "RemoveContainer" containerID="50e769eaad0e9eeeee0b19c6cfef458c6f71c0bba90bcec826c96e3225a5eb81"
Nov 03 18:15:20.064503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.064479    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn2ml\" (UniqueName: \"kubernetes.io/projected/6784968f-f0cc-4bc2-a870-4e42548bf837-kube-api-access-xn2ml\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.065749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.065715    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r" event={"ID":"6784968f-f0cc-4bc2-a870-4e42548bf837","Type":"ContainerDied","Data":"8cdf49cc5de9f56f6c86b9551a05e6b1acee17051d4ed3fd0bfab0901c72cf5d"}
Nov 03 18:15:20.065850 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.065720    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"
Nov 03 18:15:20.074026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.074010    4493 scope.go:122] "RemoveContainer" containerID="087d4c033c8b2f42a1c9837d42081c57c64c9ec9ae6bc40537afb1dc7d1b12c8"
Nov 03 18:15:20.180183 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.180154    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"]
Nov 03 18:15:20.242201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.242073    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"]
Nov 03 18:15:20.246685 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.246648    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-43bdc0-29895495-mfw9h"]
Nov 03 18:15:20.353526 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.353503    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:20.355491 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.355471    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"]
Nov 03 18:15:20.362010 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.361987    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c76280-29895495-vrm6r"]
Nov 03 18:15:20.469623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.469592    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a92c0a-f5cd-4d86-9544-4129e2860646-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a92c0a-f5cd-4d86-9544-4129e2860646-maas-api-service-ca\") pod \"99a92c0a-f5cd-4d86-9544-4129e2860646\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") "
Nov 03 18:15:20.469776 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.469751    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a92c0a-f5cd-4d86-9544-4129e2860646-kube-api-access-62fkb\" (UniqueName: \"kubernetes.io/projected/99a92c0a-f5cd-4d86-9544-4129e2860646-kube-api-access-62fkb\") pod \"99a92c0a-f5cd-4d86-9544-4129e2860646\" (UID: \"99a92c0a-f5cd-4d86-9544-4129e2860646\") "
Nov 03 18:15:20.469948 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.469927    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a92c0a-f5cd-4d86-9544-4129e2860646-maas-api-service-ca" pod "99a92c0a-f5cd-4d86-9544-4129e2860646" (UID: "99a92c0a-f5cd-4d86-9544-4129e2860646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:20.470304 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.470283    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a92c0a-f5cd-4d86-9544-4129e2860646-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.472789 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.472761    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a92c0a-f5cd-4d86-9544-4129e2860646-kube-api-access-62fkb" pod "99a92c0a-f5cd-4d86-9544-4129e2860646" (UID: "99a92c0a-f5cd-4d86-9544-4129e2860646"). InnerVolumeSpecName "kube-api-access-62fkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:20.571746 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.571657    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62fkb\" (UniqueName: \"kubernetes.io/projected/99a92c0a-f5cd-4d86-9544-4129e2860646-kube-api-access-62fkb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:20.712660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.712629    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49352363-7b25-4daa-8986-744f4e269cf5" path="/var/lib/kubelet/pods/49352363-7b25-4daa-8986-744f4e269cf5/volumes"
Nov 03 18:15:20.713127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.713112    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56cc8ddf-6721-428b-972c-68a77603096a" path="/var/lib/kubelet/pods/56cc8ddf-6721-428b-972c-68a77603096a/volumes"
Nov 03 18:15:20.713462 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.713450    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" path="/var/lib/kubelet/pods/6784968f-f0cc-4bc2-a870-4e42548bf837/volumes"
Nov 03 18:15:20.714148 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:20.714132    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" path="/var/lib/kubelet/pods/6bd4081a-942d-4c14-95ee-39a7ada193b1/volumes"
Nov 03 18:15:21.074429 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:21.074393    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"
Nov 03 18:15:21.074429 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:21.074409    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n" event={"ID":"99a92c0a-f5cd-4d86-9544-4129e2860646","Type":"ContainerDied","Data":"d2e0d6a4fccd526ee082f62d71d9f4efaf66b1918103f3236fdd87d2ef312a19"}
Nov 03 18:15:21.075002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:21.074464    4493 scope.go:122] "RemoveContainer" containerID="a28a6316c8a65f1e105ff8466dee61c74b12727fb90bb9e1aa4c2ce573a91518"
Nov 03 18:15:21.243887 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:21.243839    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"]
Nov 03 18:15:21.294569 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:21.294536    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-548ca1-29895495-2jf4n"]
Nov 03 18:15:22.711921 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:22.711888    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" path="/var/lib/kubelet/pods/99a92c0a-f5cd-4d86-9544-4129e2860646/volumes"
Nov 03 18:15:28.824445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:28.824408    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" podUID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:15:29.991821 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:29.991758    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 18:15:30.072261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.072228    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-kube-api-access-qcqkf\" (UniqueName: \"kubernetes.io/projected/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-kube-api-access-qcqkf\") pod \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") "
Nov 03 18:15:30.072428 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.072365    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-plugins-config-volume\") pod \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\" (UID: \"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7\") "
Nov 03 18:15:30.072701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.072678    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-plugins-config-volume" pod "9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" (UID: "9c0e42b5-2d61-4ddc-b1e8-2230abf994c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:30.074456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.074435    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-kube-api-access-qcqkf" pod "9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" (UID: "9c0e42b5-2d61-4ddc-b1e8-2230abf994c7"). InnerVolumeSpecName "kube-api-access-qcqkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:30.139670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.139583    4493 generic.go:356] "Generic (PLEG): container finished" podID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" containerID="3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa" exitCode=0
Nov 03 18:15:30.139670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.139663    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"
Nov 03 18:15:30.139988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.139753    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" event={"ID":"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7","Type":"ContainerDied","Data":"3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa"}
Nov 03 18:15:30.139988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.139850    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj" event={"ID":"9c0e42b5-2d61-4ddc-b1e8-2230abf994c7","Type":"ContainerDied","Data":"075afb8a7ca8cf101c98e64d76a3b1c5136430b73c8c24b583e3810c6888e3e4"}
Nov 03 18:15:30.139988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.139908    4493 scope.go:122] "RemoveContainer" containerID="3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa"
Nov 03 18:15:30.160158 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.159852    4493 scope.go:122] "RemoveContainer" containerID="3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa"
Nov 03 18:15:30.160461 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:30.160378    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa\": container with ID starting with 3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa not found: ID does not exist" containerID="3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa"
Nov 03 18:15:30.160461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.160415    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa"} err="failed to get container status \"3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa\": rpc error: code = NotFound desc = could not find container \"3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa\": container with ID starting with 3f21d1f0f0cfda55dd1e6959b63a5c90082e104db108f4196af968e11406bbaa not found: ID does not exist"
Nov 03 18:15:30.190632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.190598    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqkf\" (UniqueName: \"kubernetes.io/projected/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-kube-api-access-qcqkf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:30.190888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.190850    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:30.313428 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.313342    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"]
Nov 03 18:15:30.319106 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.319080    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-548ca1-665858fdb5-md2qj"]
Nov 03 18:15:30.712582 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:30.712561    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" path="/var/lib/kubelet/pods/9c0e42b5-2d61-4ddc-b1e8-2230abf994c7/volumes"
Nov 03 18:15:33.087687 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:33.087045    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rww5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:15:33.087687 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:33.087182    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:15:36.992691 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:15:36.992644    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:15:42.364695 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:42.364662    4493 scope.go:122] "RemoveContainer" containerID="fffec639da95e83bf665e49a5bc9f65b66f26c3d461890c1f6c200e4a15fc71e"
Nov 03 18:15:42.374253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:42.374234    4493 scope.go:122] "RemoveContainer" containerID="505a237f5caa4f606ad123ee0998fdce926310228d9084776460d63bd232a4ec"
Nov 03 18:15:49.526026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.525988    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"]
Nov 03 18:15:49.526593 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526575    4493 cpu_manager.go:414] "containerMap: removing container" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.526640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526595    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.526640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526608    4493 cpu_manager.go:414] "containerMap: removing container" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.526640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526613    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.526640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526622    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main"
Nov 03 18:15:49.526640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526628    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main"
Nov 03 18:15:49.526640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526634    4493 cpu_manager.go:414] "containerMap: removing container" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.526640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526639    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526647    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526652    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526658    4493 cpu_manager.go:414] "containerMap: removing container" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526662    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526673    4493 cpu_manager.go:414] "containerMap: removing container" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526677    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526685    4493 cpu_manager.go:414] "containerMap: removing container" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526690    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526696    4493 cpu_manager.go:414] "containerMap: removing container" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526703    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526710    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526714    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526723    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526727    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526733    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526737    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526742    4493 cpu_manager.go:414] "containerMap: removing container" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526746    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526753    4493 cpu_manager.go:414] "containerMap: removing container" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526757    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526766    4493 cpu_manager.go:414] "containerMap: removing container" podUID="56cc8ddf-6721-428b-972c-68a77603096a" containerName="payload-processing"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526770    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56cc8ddf-6721-428b-972c-68a77603096a" containerName="payload-processing"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526777    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526781    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526789    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526793    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526800    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526805    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526816    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" containerName="payload-processing"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526820    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" containerName="payload-processing"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526826    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d7c5412f-fb67-47ac-91b5-bff1a6561a2c" containerName="revoke-keys"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526830    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c5412f-fb67-47ac-91b5-bff1a6561a2c" containerName="revoke-keys"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526837    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526841    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526853    4493 cpu_manager.go:414] "containerMap: removing container" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.526859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526857    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526950    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526957    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526963    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526970    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="381f0f27-9512-4d8c-a529-5b8cda13dc7f" containerName="main"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526976    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="56cc8ddf-6721-428b-972c-68a77603096a" containerName="payload-processing"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526982    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526989    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.526995    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527001    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0e42b5-2d61-4ddc-b1e8-2230abf994c7" containerName="payload-processing"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527007    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c5412f-fb67-47ac-91b5-bff1a6561a2c" containerName="revoke-keys"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527014    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527019    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527024    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527029    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66cfe66-7c3d-423c-b71e-b28fa4e47d75" containerName="main"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527034    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527038    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527043    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f9f592-5d8b-4cd3-9883-aef79b463eca" containerName="istio-proxy"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527049    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="503a8b5b-5bf6-4f06-8268-3cda7c14b9ae" containerName="main"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527054    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="809fc6c4-b67d-4d4e-9225-39b4a2a6757d" containerName="main"
Nov 03 18:15:49.527998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.527060    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="07be7d18-9942-4d43-889a-0cfe69b59bb1" containerName="istio-proxy"
Nov 03 18:15:49.532167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.532147    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.565473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.565449    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/a76421f5-61b3-460a-b26b-9f980cfdc9aa-kube-api-access-6zr6r\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.565601 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.565579    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.565647 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.565623    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.638272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.638238    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:15:49.638272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.638272    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:15:49.638539 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.638287    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:15:49.638539 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.638426    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:15:49.641062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.640994    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"]
Nov 03 18:15:49.641524 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641506    4493 cpu_manager.go:414] "containerMap: removing container" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.641524 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641523    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49352363-7b25-4daa-8986-744f4e269cf5" containerName="cleanup"
Nov 03 18:15:49.641700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641545    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.641700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641550    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.641700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641652    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e432deb9-c69b-4766-b635-b6b6baee7b1a" containerName="cleanup"
Nov 03 18:15:49.641700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641674    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6784968f-f0cc-4bc2-a870-4e42548bf837" containerName="cleanup"
Nov 03 18:15:49.641700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641684    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bd4081a-942d-4c14-95ee-39a7ada193b1" containerName="cleanup"
Nov 03 18:15:49.641700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.641692    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a92c0a-f5cd-4d86-9544-4129e2860646" containerName="cleanup"
Nov 03 18:15:49.645047 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.645027    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"]
Nov 03 18:15:49.645166 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.645112    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.660256 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.660229    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"]
Nov 03 18:15:49.666977 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.666949    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.667089 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.666995    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8pt\" (UniqueName: \"kubernetes.io/projected/8175b8ed-761b-4a02-997d-fc9c2871640e-kube-api-access-jm8pt\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.667089 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.667036    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.667222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.667198    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/a76421f5-61b3-460a-b26b-9f980cfdc9aa-kube-api-access-6zr6r\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.667283 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.667267    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.667336 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.667306    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.667752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.667727    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.669606 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.669580    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.730642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.730613    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/a76421f5-61b3-460a-b26b-9f980cfdc9aa-kube-api-access-6zr6r\") pod \"maas-api-revoke-sub-9f1b8c74-m8vmv\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.768520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.768483    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.768661 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.768554    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.768724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.768688    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8pt\" (UniqueName: \"kubernetes.io/projected/8175b8ed-761b-4a02-997d-fc9c2871640e-kube-api-access-jm8pt\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.769241 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.769184    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.771206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.771179    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.832103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.832039    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8pt\" (UniqueName: \"kubernetes.io/projected/8175b8ed-761b-4a02-997d-fc9c2871640e-kube-api-access-jm8pt\") pod \"maas-api-revoke-sub-a4b18fbf-drq9m\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:49.844924 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.844901    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:49.964617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:49.964583    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:50.222737 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:50.222708    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"]
Nov 03 18:15:50.223915 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:50.223892    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76421f5_61b3_460a_b26b_9f980cfdc9aa.slice/crio-50cfeddd1996730fe2656a7cee1efa85c4d53a626dffb341fc5f25c37904dab2 WatchSource:0}: Error finding container 50cfeddd1996730fe2656a7cee1efa85c4d53a626dffb341fc5f25c37904dab2: Status 404 returned error can't find the container with id 50cfeddd1996730fe2656a7cee1efa85c4d53a626dffb341fc5f25c37904dab2
Nov 03 18:15:50.309649 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:50.309617    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv" event={"ID":"a76421f5-61b3-460a-b26b-9f980cfdc9aa","Type":"ContainerStarted","Data":"50cfeddd1996730fe2656a7cee1efa85c4d53a626dffb341fc5f25c37904dab2"}
Nov 03 18:15:50.341720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:50.341693    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"]
Nov 03 18:15:50.343195 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:50.343165    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8175b8ed_761b_4a02_997d_fc9c2871640e.slice/crio-2c4d9bbdcb94dce751e8904c7321b29f1f18d220833606e0141433b5e1b721db WatchSource:0}: Error finding container 2c4d9bbdcb94dce751e8904c7321b29f1f18d220833606e0141433b5e1b721db: Status 404 returned error can't find the container with id 2c4d9bbdcb94dce751e8904c7321b29f1f18d220833606e0141433b5e1b721db
Nov 03 18:15:51.321076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:51.321043    4493 generic.go:356] "Generic (PLEG): container finished" podID="a76421f5-61b3-460a-b26b-9f980cfdc9aa" containerID="85f2635b9d800c6627593e11c6f2e3161115526ce7b489a5649b53e3de78f9d5" exitCode=0
Nov 03 18:15:51.321452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:51.321126    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv" event={"ID":"a76421f5-61b3-460a-b26b-9f980cfdc9aa","Type":"ContainerDied","Data":"85f2635b9d800c6627593e11c6f2e3161115526ce7b489a5649b53e3de78f9d5"}
Nov 03 18:15:51.322993 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:51.322960    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m" event={"ID":"8175b8ed-761b-4a02-997d-fc9c2871640e","Type":"ContainerStarted","Data":"4695d3c859fb723cb584b1f71700d8469e2819154f5454a22e33363e7080fc64"}
Nov 03 18:15:51.323154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:51.323015    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m" event={"ID":"8175b8ed-761b-4a02-997d-fc9c2871640e","Type":"ContainerStarted","Data":"2c4d9bbdcb94dce751e8904c7321b29f1f18d220833606e0141433b5e1b721db"}
Nov 03 18:15:51.564577 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:51.564487    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m" podStartSLOduration=2.564473475 podStartE2EDuration="2.564473475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:51.564117063 +0000 UTC m=+2531.023288060" watchObservedRunningTime="2026-11-03 18:15:51.564473475 +0000 UTC m=+2531.023644455"
Nov 03 18:15:52.331742 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.331705    4493 generic.go:356] "Generic (PLEG): container finished" podID="8175b8ed-761b-4a02-997d-fc9c2871640e" containerID="4695d3c859fb723cb584b1f71700d8469e2819154f5454a22e33363e7080fc64" exitCode=0
Nov 03 18:15:52.332236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.331792    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m" event={"ID":"8175b8ed-761b-4a02-997d-fc9c2871640e","Type":"ContainerDied","Data":"4695d3c859fb723cb584b1f71700d8469e2819154f5454a22e33363e7080fc64"}
Nov 03 18:15:52.475920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.475865    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:52.603319 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.603242    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76421f5-61b3-460a-b26b-9f980cfdc9aa-kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/a76421f5-61b3-460a-b26b-9f980cfdc9aa-kube-api-access-6zr6r\") pod \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") "
Nov 03 18:15:52.603319 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.603289    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-openshift-ca\") pod \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") "
Nov 03 18:15:52.603549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.603440    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-xks-ca\") pod \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\" (UID: \"a76421f5-61b3-460a-b26b-9f980cfdc9aa\") "
Nov 03 18:15:52.603685 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.603658    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-openshift-ca" pod "a76421f5-61b3-460a-b26b-9f980cfdc9aa" (UID: "a76421f5-61b3-460a-b26b-9f980cfdc9aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:52.604006 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.603982    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:52.605700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.605661    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-xks-ca" pod "a76421f5-61b3-460a-b26b-9f980cfdc9aa" (UID: "a76421f5-61b3-460a-b26b-9f980cfdc9aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:52.612031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.612000    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76421f5-61b3-460a-b26b-9f980cfdc9aa-kube-api-access-6zr6r" pod "a76421f5-61b3-460a-b26b-9f980cfdc9aa" (UID: "a76421f5-61b3-460a-b26b-9f980cfdc9aa"). InnerVolumeSpecName "kube-api-access-6zr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:52.705560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.705531    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zr6r\" (UniqueName: \"kubernetes.io/projected/a76421f5-61b3-460a-b26b-9f980cfdc9aa-kube-api-access-6zr6r\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:52.705705 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:52.705668    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76421f5-61b3-460a-b26b-9f980cfdc9aa-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.339603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.339575    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"
Nov 03 18:15:53.339603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.339595    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv" event={"ID":"a76421f5-61b3-460a-b26b-9f980cfdc9aa","Type":"ContainerDied","Data":"50cfeddd1996730fe2656a7cee1efa85c4d53a626dffb341fc5f25c37904dab2"}
Nov 03 18:15:53.340188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.339664    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50cfeddd1996730fe2656a7cee1efa85c4d53a626dffb341fc5f25c37904dab2"
Nov 03 18:15:53.473447 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.473422    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:53.514387 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.514364    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8175b8ed-761b-4a02-997d-fc9c2871640e-kube-api-access-jm8pt\" (UniqueName: \"kubernetes.io/projected/8175b8ed-761b-4a02-997d-fc9c2871640e-kube-api-access-jm8pt\") pod \"8175b8ed-761b-4a02-997d-fc9c2871640e\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") "
Nov 03 18:15:53.514522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.514501    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-openshift-ca\") pod \"8175b8ed-761b-4a02-997d-fc9c2871640e\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") "
Nov 03 18:15:53.514583 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.514571    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-xks-ca\") pod \"8175b8ed-761b-4a02-997d-fc9c2871640e\" (UID: \"8175b8ed-761b-4a02-997d-fc9c2871640e\") "
Nov 03 18:15:53.514837 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.514813    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-openshift-ca" pod "8175b8ed-761b-4a02-997d-fc9c2871640e" (UID: "8175b8ed-761b-4a02-997d-fc9c2871640e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:53.516517 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.516495    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8175b8ed-761b-4a02-997d-fc9c2871640e-kube-api-access-jm8pt" pod "8175b8ed-761b-4a02-997d-fc9c2871640e" (UID: "8175b8ed-761b-4a02-997d-fc9c2871640e"). InnerVolumeSpecName "kube-api-access-jm8pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:53.516635 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.516616    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-xks-ca" pod "8175b8ed-761b-4a02-997d-fc9c2871640e" (UID: "8175b8ed-761b-4a02-997d-fc9c2871640e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:53.615996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.615907    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.615996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.615948    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b8ed-761b-4a02-997d-fc9c2871640e-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:53.615996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:53.615966    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm8pt\" (UniqueName: \"kubernetes.io/projected/8175b8ed-761b-4a02-997d-fc9c2871640e-kube-api-access-jm8pt\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:54.363900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:54.363171    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m" event={"ID":"8175b8ed-761b-4a02-997d-fc9c2871640e","Type":"ContainerDied","Data":"2c4d9bbdcb94dce751e8904c7321b29f1f18d220833606e0141433b5e1b721db"}
Nov 03 18:15:54.363900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:54.363235    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c4d9bbdcb94dce751e8904c7321b29f1f18d220833606e0141433b5e1b721db"
Nov 03 18:15:54.363900 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:54.363308    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"
Nov 03 18:15:55.200673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.200634    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"]
Nov 03 18:15:55.201338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.201317    4493 cpu_manager.go:414] "containerMap: removing container" podUID="8175b8ed-761b-4a02-997d-fc9c2871640e" containerName="revoke-keys"
Nov 03 18:15:55.201389 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.201343    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8175b8ed-761b-4a02-997d-fc9c2871640e" containerName="revoke-keys"
Nov 03 18:15:55.201389 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.201375    4493 cpu_manager.go:414] "containerMap: removing container" podUID="a76421f5-61b3-460a-b26b-9f980cfdc9aa" containerName="revoke-keys"
Nov 03 18:15:55.201389 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.201383    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76421f5-61b3-460a-b26b-9f980cfdc9aa" containerName="revoke-keys"
Nov 03 18:15:55.201523 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.201509    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76421f5-61b3-460a-b26b-9f980cfdc9aa" containerName="revoke-keys"
Nov 03 18:15:55.201559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.201538    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8175b8ed-761b-4a02-997d-fc9c2871640e" containerName="revoke-keys"
Nov 03 18:15:55.205266 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.205246    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.247822 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.247794    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.248035 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.248012    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.248120 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.248095    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5c2\" (UniqueName: \"kubernetes.io/projected/17ae6c75-4276-4216-b218-03a575bc1e7e-kube-api-access-nv5c2\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.260229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.260202    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"]
Nov 03 18:15:55.263800 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.263781    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.308239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.308209    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:15:55.308239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.308226    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:15:55.308487 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.308332    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:15:55.310830 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.310806    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:15:55.312746 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.312717    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"]
Nov 03 18:15:55.324118 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.324091    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"]
Nov 03 18:15:55.348985 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.348955    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.349159 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.349000    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mjf\" (UniqueName: \"kubernetes.io/projected/3fc872fd-ab89-4113-8e61-46d9e63902c7-kube-api-access-72mjf\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.349159 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.349039    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.349159 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.349154    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.349346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.349184    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.349346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.349227    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5c2\" (UniqueName: \"kubernetes.io/projected/17ae6c75-4276-4216-b218-03a575bc1e7e-kube-api-access-nv5c2\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.349766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.349739    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.351725 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.351704    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.412935 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.412902    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5c2\" (UniqueName: \"kubernetes.io/projected/17ae6c75-4276-4216-b218-03a575bc1e7e-kube-api-access-nv5c2\") pod \"maas-api-revoke-sub-634be419-c5xnc\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.450053 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.450023    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mjf\" (UniqueName: \"kubernetes.io/projected/3fc872fd-ab89-4113-8e61-46d9e63902c7-kube-api-access-72mjf\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.450198 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.450077    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.450198 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.450146    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.451060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.451003    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.452492 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.452463    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.513606 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.513571    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mjf\" (UniqueName: \"kubernetes.io/projected/3fc872fd-ab89-4113-8e61-46d9e63902c7-kube-api-access-72mjf\") pod \"maas-api-revoke-sub-c0346dc1-vs2br\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.518423 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.518398    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:55.582865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.582615    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:55.896392 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:55.896364    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"]
Nov 03 18:15:55.897632 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:55.897607    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ae6c75_4276_4216_b218_03a575bc1e7e.slice/crio-a92f737e31026ddb6fff15207854a7c4965be1a46aacbfd4e5b3d9a729701719 WatchSource:0}: Error finding container a92f737e31026ddb6fff15207854a7c4965be1a46aacbfd4e5b3d9a729701719: Status 404 returned error can't find the container with id a92f737e31026ddb6fff15207854a7c4965be1a46aacbfd4e5b3d9a729701719
Nov 03 18:15:56.000318 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:56.000294    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"]
Nov 03 18:15:56.001616 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:15:56.001591    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc872fd_ab89_4113_8e61_46d9e63902c7.slice/crio-71d83c33a53b3aedca7e2236f1cddbda6f29e4f06f3543d99de5a47866d5645e WatchSource:0}: Error finding container 71d83c33a53b3aedca7e2236f1cddbda6f29e4f06f3543d99de5a47866d5645e: Status 404 returned error can't find the container with id 71d83c33a53b3aedca7e2236f1cddbda6f29e4f06f3543d99de5a47866d5645e
Nov 03 18:15:56.378978 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:56.378945    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br" event={"ID":"3fc872fd-ab89-4113-8e61-46d9e63902c7","Type":"ContainerStarted","Data":"04c2389093ce7d18b9933516bf03cb24ecc8217f8c1250cb224a952761867cfd"}
Nov 03 18:15:56.379153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:56.378995    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br" event={"ID":"3fc872fd-ab89-4113-8e61-46d9e63902c7","Type":"ContainerStarted","Data":"71d83c33a53b3aedca7e2236f1cddbda6f29e4f06f3543d99de5a47866d5645e"}
Nov 03 18:15:56.380496 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:56.380471    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc" event={"ID":"17ae6c75-4276-4216-b218-03a575bc1e7e","Type":"ContainerStarted","Data":"da32e40c7399124ac45d013c149b9e138655e58c73516b8d8ac6fa5b575abcbf"}
Nov 03 18:15:56.380611 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:56.380521    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc" event={"ID":"17ae6c75-4276-4216-b218-03a575bc1e7e","Type":"ContainerStarted","Data":"a92f737e31026ddb6fff15207854a7c4965be1a46aacbfd4e5b3d9a729701719"}
Nov 03 18:15:56.501228 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:56.501112    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br" podStartSLOduration=1.501095146 podStartE2EDuration="1.501095146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:56.498543671 +0000 UTC m=+2535.957714659" watchObservedRunningTime="2026-11-03 18:15:56.501095146 +0000 UTC m=+2535.960266137"
Nov 03 18:15:56.620115 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:56.620062    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc" podStartSLOduration=1.620047225 podStartE2EDuration="1.620047225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:56.618420053 +0000 UTC m=+2536.077591043" watchObservedRunningTime="2026-11-03 18:15:56.620047225 +0000 UTC m=+2536.079218211"
Nov 03 18:15:57.387778 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:57.387741    4493 generic.go:356] "Generic (PLEG): container finished" podID="3fc872fd-ab89-4113-8e61-46d9e63902c7" containerID="04c2389093ce7d18b9933516bf03cb24ecc8217f8c1250cb224a952761867cfd" exitCode=0
Nov 03 18:15:57.388008 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:57.387830    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br" event={"ID":"3fc872fd-ab89-4113-8e61-46d9e63902c7","Type":"ContainerDied","Data":"04c2389093ce7d18b9933516bf03cb24ecc8217f8c1250cb224a952761867cfd"}
Nov 03 18:15:57.389537 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:57.389508    4493 generic.go:356] "Generic (PLEG): container finished" podID="17ae6c75-4276-4216-b218-03a575bc1e7e" containerID="da32e40c7399124ac45d013c149b9e138655e58c73516b8d8ac6fa5b575abcbf" exitCode=0
Nov 03 18:15:57.389675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:57.389543    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc" event={"ID":"17ae6c75-4276-4216-b218-03a575bc1e7e","Type":"ContainerDied","Data":"da32e40c7399124ac45d013c149b9e138655e58c73516b8d8ac6fa5b575abcbf"}
Nov 03 18:15:58.540154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.540129    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:58.583585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.583562    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:58.584754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.584736    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-openshift-ca\") pod \"17ae6c75-4276-4216-b218-03a575bc1e7e\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") "
Nov 03 18:15:58.584838 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.584822    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-xks-ca\") pod \"17ae6c75-4276-4216-b218-03a575bc1e7e\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") "
Nov 03 18:15:58.585033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.585015    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ae6c75-4276-4216-b218-03a575bc1e7e-kube-api-access-nv5c2\" (UniqueName: \"kubernetes.io/projected/17ae6c75-4276-4216-b218-03a575bc1e7e-kube-api-access-nv5c2\") pod \"17ae6c75-4276-4216-b218-03a575bc1e7e\" (UID: \"17ae6c75-4276-4216-b218-03a575bc1e7e\") "
Nov 03 18:15:58.585136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.585119    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-openshift-ca" pod "17ae6c75-4276-4216-b218-03a575bc1e7e" (UID: "17ae6c75-4276-4216-b218-03a575bc1e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:58.585414 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.585385    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:58.586920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.586896    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-xks-ca" pod "17ae6c75-4276-4216-b218-03a575bc1e7e" (UID: "17ae6c75-4276-4216-b218-03a575bc1e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:58.587056 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.587039    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ae6c75-4276-4216-b218-03a575bc1e7e-kube-api-access-nv5c2" pod "17ae6c75-4276-4216-b218-03a575bc1e7e" (UID: "17ae6c75-4276-4216-b218-03a575bc1e7e"). InnerVolumeSpecName "kube-api-access-nv5c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:58.686161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.686076    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-openshift-ca\") pod \"3fc872fd-ab89-4113-8e61-46d9e63902c7\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") "
Nov 03 18:15:58.686161 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.686134    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-xks-ca\") pod \"3fc872fd-ab89-4113-8e61-46d9e63902c7\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") "
Nov 03 18:15:58.686379 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.686216    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc872fd-ab89-4113-8e61-46d9e63902c7-kube-api-access-72mjf\" (UniqueName: \"kubernetes.io/projected/3fc872fd-ab89-4113-8e61-46d9e63902c7-kube-api-access-72mjf\") pod \"3fc872fd-ab89-4113-8e61-46d9e63902c7\" (UID: \"3fc872fd-ab89-4113-8e61-46d9e63902c7\") "
Nov 03 18:15:58.686602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.686578    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ae6c75-4276-4216-b218-03a575bc1e7e-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:58.686602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.686604    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv5c2\" (UniqueName: \"kubernetes.io/projected/17ae6c75-4276-4216-b218-03a575bc1e7e-kube-api-access-nv5c2\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:58.687113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.686775    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-openshift-ca" pod "3fc872fd-ab89-4113-8e61-46d9e63902c7" (UID: "3fc872fd-ab89-4113-8e61-46d9e63902c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:15:58.689034 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.689004    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-xks-ca" pod "3fc872fd-ab89-4113-8e61-46d9e63902c7" (UID: "3fc872fd-ab89-4113-8e61-46d9e63902c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:15:58.689034 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.689015    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc872fd-ab89-4113-8e61-46d9e63902c7-kube-api-access-72mjf" pod "3fc872fd-ab89-4113-8e61-46d9e63902c7" (UID: "3fc872fd-ab89-4113-8e61-46d9e63902c7"). InnerVolumeSpecName "kube-api-access-72mjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:15:58.788066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.788012    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mjf\" (UniqueName: \"kubernetes.io/projected/3fc872fd-ab89-4113-8e61-46d9e63902c7-kube-api-access-72mjf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:58.788066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.788060    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:58.788273 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:58.788077    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc872fd-ab89-4113-8e61-46d9e63902c7-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:15:59.402136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:59.402105    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"
Nov 03 18:15:59.402351 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:59.402110    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br" event={"ID":"3fc872fd-ab89-4113-8e61-46d9e63902c7","Type":"ContainerDied","Data":"71d83c33a53b3aedca7e2236f1cddbda6f29e4f06f3543d99de5a47866d5645e"}
Nov 03 18:15:59.402351 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:59.402243    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71d83c33a53b3aedca7e2236f1cddbda6f29e4f06f3543d99de5a47866d5645e"
Nov 03 18:15:59.403663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:59.403643    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"
Nov 03 18:15:59.403779 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:59.403688    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc" event={"ID":"17ae6c75-4276-4216-b218-03a575bc1e7e","Type":"ContainerDied","Data":"a92f737e31026ddb6fff15207854a7c4965be1a46aacbfd4e5b3d9a729701719"}
Nov 03 18:15:59.403779 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:15:59.403729    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a92f737e31026ddb6fff15207854a7c4965be1a46aacbfd4e5b3d9a729701719"
Nov 03 18:16:01.748982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.748950    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"]
Nov 03 18:16:01.749380 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.749367    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3fc872fd-ab89-4113-8e61-46d9e63902c7" containerName="revoke-keys"
Nov 03 18:16:01.749431 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.749380    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc872fd-ab89-4113-8e61-46d9e63902c7" containerName="revoke-keys"
Nov 03 18:16:01.749431 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.749421    4493 cpu_manager.go:414] "containerMap: removing container" podUID="17ae6c75-4276-4216-b218-03a575bc1e7e" containerName="revoke-keys"
Nov 03 18:16:01.749431 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.749426    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ae6c75-4276-4216-b218-03a575bc1e7e" containerName="revoke-keys"
Nov 03 18:16:01.749534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.749496    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ae6c75-4276-4216-b218-03a575bc1e7e" containerName="revoke-keys"
Nov 03 18:16:01.749534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.749506    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc872fd-ab89-4113-8e61-46d9e63902c7" containerName="revoke-keys"
Nov 03 18:16:01.752862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.752841    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.827517 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827478    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.827732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827536    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.827732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827555    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvw8l\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-kube-api-access-pvw8l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.827732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827583    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.827732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827633    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.827732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827661    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.827732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827682    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929153f0-5444-407c-a543-7650ecebdc93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.828395 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827737    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929153f0-5444-407c-a543-7650ecebdc93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.828395 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.827755    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.860705 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.860674    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"]
Nov 03 18:16:01.929201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929167    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929153f0-5444-407c-a543-7650ecebdc93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929201 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929204    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929269    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929297    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929326    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvw8l\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-kube-api-access-pvw8l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929366    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929421    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929459    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929485    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929153f0-5444-407c-a543-7650ecebdc93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929669    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929754    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.929919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929795    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.930033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.929960    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.930033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.930003    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929153f0-5444-407c-a543-7650ecebdc93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.932009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.931985    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.932075 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.932036    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929153f0-5444-407c-a543-7650ecebdc93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.992708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.992678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:01.992862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:01.992832    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvw8l\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-kube-api-access-pvw8l\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q2k5n\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:02.065412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.065335    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:02.507312 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.507259    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"]
Nov 03 18:16:02.513417 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:16:02.513062    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929153f0_5444_407c_a543_7650ecebdc93.slice/crio-04d2d6c59078e1d4a8c487326be99625518a46697838cb2bd48e129ce700f66b WatchSource:0}: Error finding container 04d2d6c59078e1d4a8c487326be99625518a46697838cb2bd48e129ce700f66b: Status 404 returned error can't find the container with id 04d2d6c59078e1d4a8c487326be99625518a46697838cb2bd48e129ce700f66b
Nov 03 18:16:02.533039 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.529568    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:16:02.533362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.533328    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:16:02.533545 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.533524    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:16:02.813231 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.813142    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"]
Nov 03 18:16:02.817517 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.817491    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:02.877063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.877026    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ca3f31\""
Nov 03 18:16:02.877448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.877424    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ca3f31-dockercfg-vlgw6\""
Nov 03 18:16:02.880311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.880283    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"]
Nov 03 18:16:02.946616 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.946386    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:02.947346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.947306    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wblpz\" (UniqueName: \"kubernetes.io/projected/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-kube-api-access-wblpz\") pod \"payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:02.960573 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.958211    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"]
Nov 03 18:16:02.962953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:02.962930    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.008781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.008739    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"]
Nov 03 18:16:03.048787 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.048752    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wblpz\" (UniqueName: \"kubernetes.io/projected/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-kube-api-access-wblpz\") pod \"payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:03.048970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.048818    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72pf\" (UniqueName: \"kubernetes.io/projected/d37a77a9-db23-40a1-9472-0494d16783e4-kube-api-access-w72pf\") pod \"payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.048970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.048865    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:03.048970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.048923    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a77a9-db23-40a1-9472-0494d16783e4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.049848 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.049816    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:03.113005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.112918    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wblpz\" (UniqueName: \"kubernetes.io/projected/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-kube-api-access-wblpz\") pod \"payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:03.130461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.130435    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:03.150197 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.150169    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a77a9-db23-40a1-9472-0494d16783e4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.150375 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.150356    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72pf\" (UniqueName: \"kubernetes.io/projected/d37a77a9-db23-40a1-9472-0494d16783e4-kube-api-access-w72pf\") pod \"payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.151062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.151036    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a77a9-db23-40a1-9472-0494d16783e4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.219694 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.219662    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72pf\" (UniqueName: \"kubernetes.io/projected/d37a77a9-db23-40a1-9472-0494d16783e4-kube-api-access-w72pf\") pod \"payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") " pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.282172 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.282133    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:03.444988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.438558    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" event={"ID":"929153f0-5444-407c-a543-7650ecebdc93","Type":"ContainerStarted","Data":"71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4"}
Nov 03 18:16:03.444988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.438657    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" event={"ID":"929153f0-5444-407c-a543-7650ecebdc93","Type":"ContainerStarted","Data":"04d2d6c59078e1d4a8c487326be99625518a46697838cb2bd48e129ce700f66b"}
Nov 03 18:16:03.541074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.541032    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"]
Nov 03 18:16:03.541854 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:16:03.541819    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae3ce93_e6ea_4b75_a364_96e7c6458a63.slice/crio-9856f86703fee62e8f21aab21007a8690332e68725f1fe77962d28d7291b8345 WatchSource:0}: Error finding container 9856f86703fee62e8f21aab21007a8690332e68725f1fe77962d28d7291b8345: Status 404 returned error can't find the container with id 9856f86703fee62e8f21aab21007a8690332e68725f1fe77962d28d7291b8345
Nov 03 18:16:03.564581 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.564525    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podStartSLOduration=2.564506881 podStartE2EDuration="2.564506881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:03.564407487 +0000 UTC m=+2543.023578487" watchObservedRunningTime="2026-11-03 18:16:03.564506881 +0000 UTC m=+2543.023677868"
Nov 03 18:16:03.687159 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:03.687128    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"]
Nov 03 18:16:03.688177 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:16:03.688151    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37a77a9_db23_40a1_9472_0494d16783e4.slice/crio-1b1462ddaa7fd68c85af6a06afb4ef8d5733a3a7f1d82f06d9d5275ca62c52a5 WatchSource:0}: Error finding container 1b1462ddaa7fd68c85af6a06afb4ef8d5733a3a7f1d82f06d9d5275ca62c52a5: Status 404 returned error can't find the container with id 1b1462ddaa7fd68c85af6a06afb4ef8d5733a3a7f1d82f06d9d5275ca62c52a5
Nov 03 18:16:04.066608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.066561    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:04.068452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.068422    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q2k5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 18:16:04.068636 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.068475    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 18:16:04.450465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.450369    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9" event={"ID":"d37a77a9-db23-40a1-9472-0494d16783e4","Type":"ContainerStarted","Data":"3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea"}
Nov 03 18:16:04.450465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.450437    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9" event={"ID":"d37a77a9-db23-40a1-9472-0494d16783e4","Type":"ContainerStarted","Data":"1b1462ddaa7fd68c85af6a06afb4ef8d5733a3a7f1d82f06d9d5275ca62c52a5"}
Nov 03 18:16:04.450465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.450461    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:04.452104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.452073    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8" event={"ID":"6ae3ce93-e6ea-4b75-a364-96e7c6458a63","Type":"ContainerStarted","Data":"558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af"}
Nov 03 18:16:04.452236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.452121    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8" event={"ID":"6ae3ce93-e6ea-4b75-a364-96e7c6458a63","Type":"ContainerStarted","Data":"9856f86703fee62e8f21aab21007a8690332e68725f1fe77962d28d7291b8345"}
Nov 03 18:16:04.452288 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.452269    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:04.571340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.571283    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9" podStartSLOduration=2.5712712509999998 podStartE2EDuration="2.571271251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:04.570782982 +0000 UTC m=+2544.029953986" watchObservedRunningTime="2026-11-03 18:16:04.571271251 +0000 UTC m=+2544.030442237"
Nov 03 18:16:04.705630 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:04.705513    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8" podStartSLOduration=2.705499477 podStartE2EDuration="2.705499477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:04.703535491 +0000 UTC m=+2544.162706476" watchObservedRunningTime="2026-11-03 18:16:04.705499477 +0000 UTC m=+2544.164670459"
Nov 03 18:16:05.066645 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:05.066609    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q2k5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 18:16:05.067092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:05.066669    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 18:16:06.066563 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:06.066529    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q2k5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 18:16:06.066738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:06.066597    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 18:16:06.992265 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:16:06.992223    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:16:07.066102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.066068    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q2k5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 18:16:07.066319 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.066130    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 18:16:07.857541 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.857511    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"]
Nov 03 18:16:07.861116 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.861097    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:07.944260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.939299    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9pxb\" (UniqueName: \"kubernetes.io/projected/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-kube-api-access-b9pxb\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:07.944260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.939388    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:07.944260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.939491    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:07.944260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.944161    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"]
Nov 03 18:16:07.973154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.973114    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:16:07.973154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.973114    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:16:07.973154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.973120    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:16:07.973394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:07.973129    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:16:08.040651 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.040614    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:08.040906 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.040772    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9pxb\" (UniqueName: \"kubernetes.io/projected/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-kube-api-access-b9pxb\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:08.040906 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.040836    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:08.041629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.041598    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:08.043763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.043733    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:08.065824 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.065794    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q2k5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 18:16:08.066021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.065842    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 18:16:08.105960 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.105931    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9pxb\" (UniqueName: \"kubernetes.io/projected/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-kube-api-access-b9pxb\") pod \"maas-api-revoke-sub-6b5a56c1-qc7s7\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:08.175533 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.175451    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:08.579728 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:08.579700    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"]
Nov 03 18:16:08.581411 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:16:08.581377    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded80f6ad_e21d_41c4_9afb_6c4bcd9b31e3.slice/crio-146d5be5a70edbce588d6dcd139f06bcfa6defbf47e14b8d4aa5632ba47d985a WatchSource:0}: Error finding container 146d5be5a70edbce588d6dcd139f06bcfa6defbf47e14b8d4aa5632ba47d985a: Status 404 returned error can't find the container with id 146d5be5a70edbce588d6dcd139f06bcfa6defbf47e14b8d4aa5632ba47d985a
Nov 03 18:16:09.066582 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:09.066550    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q2k5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 18:16:09.067002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:09.066603    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 18:16:09.489046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:09.489008    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7" event={"ID":"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3","Type":"ContainerStarted","Data":"ba4dfe3be35cd2901afeeb1f89b90626c8653e5cd0880d60fc4fafb0d7ace00b"}
Nov 03 18:16:09.489203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:09.489077    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7" event={"ID":"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3","Type":"ContainerStarted","Data":"146d5be5a70edbce588d6dcd139f06bcfa6defbf47e14b8d4aa5632ba47d985a"}
Nov 03 18:16:09.610536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:09.610482    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7" podStartSLOduration=2.610467292 podStartE2EDuration="2.610467292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:09.608015109 +0000 UTC m=+2549.067186096" watchObservedRunningTime="2026-11-03 18:16:09.610467292 +0000 UTC m=+2549.069638278"
Nov 03 18:16:10.066000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:10.065960    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q2k5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 18:16:10.066211 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:10.066018    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 18:16:10.497729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:10.497686    4493 generic.go:356] "Generic (PLEG): container finished" podID="ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" containerID="ba4dfe3be35cd2901afeeb1f89b90626c8653e5cd0880d60fc4fafb0d7ace00b" exitCode=0
Nov 03 18:16:10.498270 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:10.497775    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7" event={"ID":"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3","Type":"ContainerDied","Data":"ba4dfe3be35cd2901afeeb1f89b90626c8653e5cd0880d60fc4fafb0d7ace00b"}
Nov 03 18:16:11.158048 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.158016    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:11.158811 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.158778    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:11.159194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.159176    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:16:11.654752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.654721    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:11.787758 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.787678    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-openshift-ca\") pod \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") "
Nov 03 18:16:11.787942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.787773    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-xks-ca\") pod \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") "
Nov 03 18:16:11.787942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.787848    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-kube-api-access-b9pxb\" (UniqueName: \"kubernetes.io/projected/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-kube-api-access-b9pxb\") pod \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\" (UID: \"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3\") "
Nov 03 18:16:11.788077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.788040    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-openshift-ca" pod "ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" (UID: "ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:11.788552 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.788524    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:11.790220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.790194    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-kube-api-access-b9pxb" pod "ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" (UID: "ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3"). InnerVolumeSpecName "kube-api-access-b9pxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:11.790380 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.790358    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-xks-ca" pod "ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" (UID: "ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:11.890004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.889972    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:11.890004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:11.890000    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9pxb\" (UniqueName: \"kubernetes.io/projected/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3-kube-api-access-b9pxb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:12.511014 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:12.510982    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"
Nov 03 18:16:12.511203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:12.511012    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7" event={"ID":"ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3","Type":"ContainerDied","Data":"146d5be5a70edbce588d6dcd139f06bcfa6defbf47e14b8d4aa5632ba47d985a"}
Nov 03 18:16:12.511203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:12.511087    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="146d5be5a70edbce588d6dcd139f06bcfa6defbf47e14b8d4aa5632ba47d985a"
Nov 03 18:16:15.462073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:15.462038    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:16:15.462580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:15.462127    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:16:19.166732 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.166699    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"]
Nov 03 18:16:19.167232 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.167217    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" containerName="revoke-keys"
Nov 03 18:16:19.167277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.167233    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" containerName="revoke-keys"
Nov 03 18:16:19.167346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.167336    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" containerName="revoke-keys"
Nov 03 18:16:19.170542 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.170518    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.230209 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.230179    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"]
Nov 03 18:16:19.272332 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.272297    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.272503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.272356    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8vl\" (UniqueName: \"kubernetes.io/projected/dd3294a2-6cba-4547-b528-0cf702991cdf-kube-api-access-7n8vl\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.272503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.272482    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.277027 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.277005    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:16:19.277129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.277009    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:16:19.277129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.277043    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:16:19.277207 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.277120    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:16:19.373461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.373411    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.373643 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.373499    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8vl\" (UniqueName: \"kubernetes.io/projected/dd3294a2-6cba-4547-b528-0cf702991cdf-kube-api-access-7n8vl\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.373643 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.373573    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.374223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.374197    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.375951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.375932    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.437713 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.437641    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8vl\" (UniqueName: \"kubernetes.io/projected/dd3294a2-6cba-4547-b528-0cf702991cdf-kube-api-access-7n8vl\") pod \"maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.483351 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.483318    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:19.865632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:19.865607    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"]
Nov 03 18:16:19.867099 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:16:19.867067    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3294a2_6cba_4547_b528_0cf702991cdf.slice/crio-4eff03cd1b8f6b164fe5075040312d050dad47e8d1bebb9b2a6216fcc7c7ec07 WatchSource:0}: Error finding container 4eff03cd1b8f6b164fe5075040312d050dad47e8d1bebb9b2a6216fcc7c7ec07: Status 404 returned error can't find the container with id 4eff03cd1b8f6b164fe5075040312d050dad47e8d1bebb9b2a6216fcc7c7ec07
Nov 03 18:16:20.564142 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:20.564103    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6" event={"ID":"dd3294a2-6cba-4547-b528-0cf702991cdf","Type":"ContainerStarted","Data":"7a544bd623dab4c2ed607ce1eea960e83bd32a002f33b201b05b4b08f567cc01"}
Nov 03 18:16:20.564546 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:20.564157    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6" event={"ID":"dd3294a2-6cba-4547-b528-0cf702991cdf","Type":"ContainerStarted","Data":"4eff03cd1b8f6b164fe5075040312d050dad47e8d1bebb9b2a6216fcc7c7ec07"}
Nov 03 18:16:20.686668 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:20.686597    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6" podStartSLOduration=1.686578624 podStartE2EDuration="1.686578624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:20.684540244 +0000 UTC m=+2560.143711231" watchObservedRunningTime="2026-11-03 18:16:20.686578624 +0000 UTC m=+2560.145749614"
Nov 03 18:16:21.572122 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:21.572085    4493 generic.go:356] "Generic (PLEG): container finished" podID="dd3294a2-6cba-4547-b528-0cf702991cdf" containerID="7a544bd623dab4c2ed607ce1eea960e83bd32a002f33b201b05b4b08f567cc01" exitCode=0
Nov 03 18:16:21.572578 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:21.572170    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6" event={"ID":"dd3294a2-6cba-4547-b528-0cf702991cdf","Type":"ContainerDied","Data":"7a544bd623dab4c2ed607ce1eea960e83bd32a002f33b201b05b4b08f567cc01"}
Nov 03 18:16:22.728321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.728288    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:22.817743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.817710    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-xks-ca\") pod \"dd3294a2-6cba-4547-b528-0cf702991cdf\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") "
Nov 03 18:16:22.817944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.817819    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-openshift-ca\") pod \"dd3294a2-6cba-4547-b528-0cf702991cdf\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") "
Nov 03 18:16:22.817944 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.817849    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3294a2-6cba-4547-b528-0cf702991cdf-kube-api-access-7n8vl\" (UniqueName: \"kubernetes.io/projected/dd3294a2-6cba-4547-b528-0cf702991cdf-kube-api-access-7n8vl\") pod \"dd3294a2-6cba-4547-b528-0cf702991cdf\" (UID: \"dd3294a2-6cba-4547-b528-0cf702991cdf\") "
Nov 03 18:16:22.818213 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.818190    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-openshift-ca" pod "dd3294a2-6cba-4547-b528-0cf702991cdf" (UID: "dd3294a2-6cba-4547-b528-0cf702991cdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:16:22.818316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.818299    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:22.820159 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.820134    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3294a2-6cba-4547-b528-0cf702991cdf-kube-api-access-7n8vl" pod "dd3294a2-6cba-4547-b528-0cf702991cdf" (UID: "dd3294a2-6cba-4547-b528-0cf702991cdf"). InnerVolumeSpecName "kube-api-access-7n8vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:22.820259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.820172    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-xks-ca" pod "dd3294a2-6cba-4547-b528-0cf702991cdf" (UID: "dd3294a2-6cba-4547-b528-0cf702991cdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:22.919082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.919000    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd3294a2-6cba-4547-b528-0cf702991cdf-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:22.919082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:22.919029    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n8vl\" (UniqueName: \"kubernetes.io/projected/dd3294a2-6cba-4547-b528-0cf702991cdf-kube-api-access-7n8vl\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:23.566516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:23.566476    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"]
Nov 03 18:16:23.586466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:23.586433    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"
Nov 03 18:16:23.586625 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:23.586463    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6" event={"ID":"dd3294a2-6cba-4547-b528-0cf702991cdf","Type":"ContainerDied","Data":"4eff03cd1b8f6b164fe5075040312d050dad47e8d1bebb9b2a6216fcc7c7ec07"}
Nov 03 18:16:23.586625 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:23.586514    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eff03cd1b8f6b164fe5075040312d050dad47e8d1bebb9b2a6216fcc7c7ec07"
Nov 03 18:16:23.588483 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:23.588461    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"]
Nov 03 18:16:36.992642 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:16:36.992605    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:16:38.589296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:38.589231    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main" containerID="cri-o://0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981" gracePeriod=45
Nov 03 18:16:38.613363 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:38.613315    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" containerID="cri-o://3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93" gracePeriod=45
Nov 03 18:16:38.881242 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:38.881206    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 18:16:38.910215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:38.910190    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 18:16:39.008480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008431    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-model-cache\") pod \"b6a2d2dd-25c1-423b-924a-8fa45149987a\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") "
Nov 03 18:16:39.008670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008526    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-model-cache\") pod \"d7f4752d-8f13-4214-83ea-1748907f4eb2\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") "
Nov 03 18:16:39.008670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008562    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-tmp-dir\") pod \"d7f4752d-8f13-4214-83ea-1748907f4eb2\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") "
Nov 03 18:16:39.008670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008628    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-dshm\") pod \"d7f4752d-8f13-4214-83ea-1748907f4eb2\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") "
Nov 03 18:16:39.008670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008658    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-dshm\") pod \"b6a2d2dd-25c1-423b-924a-8fa45149987a\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") "
Nov 03 18:16:39.009062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008673    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-home\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-home\") pod \"d7f4752d-8f13-4214-83ea-1748907f4eb2\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") "
Nov 03 18:16:39.009062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008705    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a2d2dd-25c1-423b-924a-8fa45149987a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d2dd-25c1-423b-924a-8fa45149987a-tls-certs\") pod \"b6a2d2dd-25c1-423b-924a-8fa45149987a\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") "
Nov 03 18:16:39.009062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008779    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-tmp-dir\") pod \"b6a2d2dd-25c1-423b-924a-8fa45149987a\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") "
Nov 03 18:16:39.009062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008838    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a2d2dd-25c1-423b-924a-8fa45149987a-kube-api-access-wz8wp\" (UniqueName: \"kubernetes.io/projected/b6a2d2dd-25c1-423b-924a-8fa45149987a-kube-api-access-wz8wp\") pod \"b6a2d2dd-25c1-423b-924a-8fa45149987a\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") "
Nov 03 18:16:39.009062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008908    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-home\") pod \"b6a2d2dd-25c1-423b-924a-8fa45149987a\" (UID: \"b6a2d2dd-25c1-423b-924a-8fa45149987a\") "
Nov 03 18:16:39.009062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.008978    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f4752d-8f13-4214-83ea-1748907f4eb2-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f4752d-8f13-4214-83ea-1748907f4eb2-tls-certs\") pod \"d7f4752d-8f13-4214-83ea-1748907f4eb2\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") "
Nov 03 18:16:39.009062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009003    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-model-cache" pod "d7f4752d-8f13-4214-83ea-1748907f4eb2" (UID: "d7f4752d-8f13-4214-83ea-1748907f4eb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.009522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009077    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f4752d-8f13-4214-83ea-1748907f4eb2-kube-api-access-cn5ps\" (UniqueName: \"kubernetes.io/projected/d7f4752d-8f13-4214-83ea-1748907f4eb2-kube-api-access-cn5ps\") pod \"d7f4752d-8f13-4214-83ea-1748907f4eb2\" (UID: \"d7f4752d-8f13-4214-83ea-1748907f4eb2\") "
Nov 03 18:16:39.009522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009254    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-home" pod "d7f4752d-8f13-4214-83ea-1748907f4eb2" (UID: "d7f4752d-8f13-4214-83ea-1748907f4eb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.009522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009398    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-tmp-dir" pod "b6a2d2dd-25c1-423b-924a-8fa45149987a" (UID: "b6a2d2dd-25c1-423b-924a-8fa45149987a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.009522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009437    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-tmp-dir" pod "d7f4752d-8f13-4214-83ea-1748907f4eb2" (UID: "d7f4752d-8f13-4214-83ea-1748907f4eb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.009723 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009634    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-home" pod "b6a2d2dd-25c1-423b-924a-8fa45149987a" (UID: "b6a2d2dd-25c1-423b-924a-8fa45149987a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.009723 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009701    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.009839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009721    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.009839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009754    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.009839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009768    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.009839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.009820    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-model-cache" pod "b6a2d2dd-25c1-423b-924a-8fa45149987a" (UID: "b6a2d2dd-25c1-423b-924a-8fa45149987a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.012317 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.012284    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a2d2dd-25c1-423b-924a-8fa45149987a-tls-certs" pod "b6a2d2dd-25c1-423b-924a-8fa45149987a" (UID: "b6a2d2dd-25c1-423b-924a-8fa45149987a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:39.012444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.012332    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-dshm" pod "b6a2d2dd-25c1-423b-924a-8fa45149987a" (UID: "b6a2d2dd-25c1-423b-924a-8fa45149987a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.013269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.013247    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f4752d-8f13-4214-83ea-1748907f4eb2-kube-api-access-cn5ps" pod "d7f4752d-8f13-4214-83ea-1748907f4eb2" (UID: "d7f4752d-8f13-4214-83ea-1748907f4eb2"). InnerVolumeSpecName "kube-api-access-cn5ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:39.013602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.013573    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-dshm" pod "d7f4752d-8f13-4214-83ea-1748907f4eb2" (UID: "d7f4752d-8f13-4214-83ea-1748907f4eb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:16:39.013689 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.013648    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a2d2dd-25c1-423b-924a-8fa45149987a-kube-api-access-wz8wp" pod "b6a2d2dd-25c1-423b-924a-8fa45149987a" (UID: "b6a2d2dd-25c1-423b-924a-8fa45149987a"). InnerVolumeSpecName "kube-api-access-wz8wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:16:39.013987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.013965    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f4752d-8f13-4214-83ea-1748907f4eb2-tls-certs" pod "d7f4752d-8f13-4214-83ea-1748907f4eb2" (UID: "d7f4752d-8f13-4214-83ea-1748907f4eb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:16:39.111348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111262    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz8wp\" (UniqueName: \"kubernetes.io/projected/b6a2d2dd-25c1-423b-924a-8fa45149987a-kube-api-access-wz8wp\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.111348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111298    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.111348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111314    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f4752d-8f13-4214-83ea-1748907f4eb2-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.111348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111329    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn5ps\" (UniqueName: \"kubernetes.io/projected/d7f4752d-8f13-4214-83ea-1748907f4eb2-kube-api-access-cn5ps\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.111348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111343    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.111733 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111357    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f4752d-8f13-4214-83ea-1748907f4eb2-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.111733 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111370    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a2d2dd-25c1-423b-924a-8fa45149987a-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.111733 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.111382    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a2d2dd-25c1-423b-924a-8fa45149987a-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:16:39.687464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.687420    4493 generic.go:356] "Generic (PLEG): container finished" podID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerID="0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981" exitCode=0
Nov 03 18:16:39.687943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.687488    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"
Nov 03 18:16:39.687943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.687495    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" event={"ID":"d7f4752d-8f13-4214-83ea-1748907f4eb2","Type":"ContainerDied","Data":"0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981"}
Nov 03 18:16:39.687943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.687539    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww" event={"ID":"d7f4752d-8f13-4214-83ea-1748907f4eb2","Type":"ContainerDied","Data":"9dc9ef29e86e5da92a54f66c4f81db99b0dea80edc8c6f141e5f99b716825e99"}
Nov 03 18:16:39.687943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.687560    4493 scope.go:122] "RemoveContainer" containerID="0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981"
Nov 03 18:16:39.689042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.689019    4493 generic.go:356] "Generic (PLEG): container finished" podID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerID="3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93" exitCode=0
Nov 03 18:16:39.689105 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.689080    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" event={"ID":"b6a2d2dd-25c1-423b-924a-8fa45149987a","Type":"ContainerDied","Data":"3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93"}
Nov 03 18:16:39.689147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.689108    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" event={"ID":"b6a2d2dd-25c1-423b-924a-8fa45149987a","Type":"ContainerDied","Data":"67aa111d178d411a1a0372955c7db46a620a045697d3acf57e422c4e28212291"}
Nov 03 18:16:39.689147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.689110    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"
Nov 03 18:16:39.698244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.698114    4493 scope.go:122] "RemoveContainer" containerID="0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981"
Nov 03 18:16:39.698464 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:16:39.698439    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981\": container with ID starting with 0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981 not found: ID does not exist" containerID="0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981"
Nov 03 18:16:39.698534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.698479    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981"} err="failed to get container status \"0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981\": rpc error: code = NotFound desc = could not find container \"0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981\": container with ID starting with 0b30ca031c627d0cc83b97f5e7113eb7585b9e1314f259ff9e8bb3de76082981 not found: ID does not exist"
Nov 03 18:16:39.698534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.698504    4493 scope.go:122] "RemoveContainer" containerID="3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93"
Nov 03 18:16:39.707653 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.707635    4493 scope.go:122] "RemoveContainer" containerID="3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93"
Nov 03 18:16:39.708560 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:16:39.708537    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93\": container with ID starting with 3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93 not found: ID does not exist" containerID="3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93"
Nov 03 18:16:39.708624 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.708570    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93"} err="failed to get container status \"3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93\": rpc error: code = NotFound desc = could not find container \"3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93\": container with ID starting with 3e7a1688e2831daa7bffaab671b85a2f1aa0c27da2dbb158f7467f52fd25db93 not found: ID does not exist"
Nov 03 18:16:39.817727 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.817686    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:16:39.862036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.861962    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"]
Nov 03 18:16:39.868270 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.868248    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/simulator-w1-c76280-kserve-6bbfd86fc4-x7ptp"]
Nov 03 18:16:39.981687 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.981654    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"]
Nov 03 18:16:39.984730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:39.984707    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-w1-c76280-kserve-dddfbf958-lsbww"]
Nov 03 18:16:40.712254 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:40.712221    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" path="/var/lib/kubelet/pods/b6a2d2dd-25c1-423b-924a-8fa45149987a/volumes"
Nov 03 18:16:40.712842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:40.712821    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" path="/var/lib/kubelet/pods/d7f4752d-8f13-4214-83ea-1748907f4eb2/volumes"
Nov 03 18:16:42.474898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:42.470893    4493 scope.go:122] "RemoveContainer" containerID="0681e025b01e816fcff678805e9b696d0300505706ee68e0aa33075fb484de4c"
Nov 03 18:16:42.508389 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:42.508364    4493 scope.go:122] "RemoveContainer" containerID="0060d75b40c25c4a4e84a8e974752565ee2595dbd74d5b25e8ce529e97b22fd0"
Nov 03 18:16:42.557920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:42.548151    4493 scope.go:122] "RemoveContainer" containerID="a4dc2563656d14bb311b7a8ae8420b0e84a350e8eeda781edf3e1e5f52ecdf04"
Nov 03 18:16:46.279286 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:46.279247    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"]
Nov 03 18:16:46.329861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:46.329822    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-548ca1-k8vhk"]
Nov 03 18:16:46.712810 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:46.712780    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed91a619-a867-4b69-8073-9b89ef8d5561" path="/var/lib/kubelet/pods/ed91a619-a867-4b69-8073-9b89ef8d5561/volumes"
Nov 03 18:16:50.295936 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.295900    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"]
Nov 03 18:16:50.296403 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296389    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main"
Nov 03 18:16:50.296448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296405    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main"
Nov 03 18:16:50.296448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296417    4493 cpu_manager.go:414] "containerMap: removing container" podUID="dd3294a2-6cba-4547-b528-0cf702991cdf" containerName="revoke-keys"
Nov 03 18:16:50.296448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296423    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3294a2-6cba-4547-b528-0cf702991cdf" containerName="revoke-keys"
Nov 03 18:16:50.296448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296434    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main"
Nov 03 18:16:50.296448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296439    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main"
Nov 03 18:16:50.296603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296535    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f4752d-8f13-4214-83ea-1748907f4eb2" containerName="main"
Nov 03 18:16:50.296603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296547    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3294a2-6cba-4547-b528-0cf702991cdf" containerName="revoke-keys"
Nov 03 18:16:50.296603 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.296555    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a2d2dd-25c1-423b-924a-8fa45149987a" containerName="main"
Nov 03 18:16:50.301763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.301744    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.359709 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.359670    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"]
Nov 03 18:16:50.360835 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.360808    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ca6b33\"/\"kube-root-ca.crt\""
Nov 03 18:16:50.407113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.407081    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ca6b33\"/\"openshift-service-ca.crt\""
Nov 03 18:16:50.407285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.407169    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ca6b33\"/\"default-dockercfg-bhkt5\""
Nov 03 18:16:50.407285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.407239    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ca6b33\"/\"test-model-ca6b33-kserve-self-signed-certs\""
Nov 03 18:16:50.418680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.418648    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tls-certs\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.418842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.418721    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-dshm\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.418919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.418835    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-model-cache\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.418975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.418957    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-home\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.419016 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.418991    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tmp-dir\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.419051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.419015    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rchb\" (UniqueName: \"kubernetes.io/projected/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-kube-api-access-4rchb\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.520441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.520398    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-dshm\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.520632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.520456    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-model-cache\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.520632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.520506    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-home\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.520632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.520528    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tmp-dir\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.520632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.520549    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rchb\" (UniqueName: \"kubernetes.io/projected/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-kube-api-access-4rchb\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.520632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.520599    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tls-certs\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.521025 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.521001    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-home\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.521090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.521026    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-model-cache\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.521090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.521055    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tmp-dir\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.522913 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.522891    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-dshm\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.523200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.523184    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tls-certs\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.587624 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.587543    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rchb\" (UniqueName: \"kubernetes.io/projected/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-kube-api-access-4rchb\") pod \"test-model-ca6b33-kserve-7fc9cfb47-fd4kl\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") " pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:50.614130 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:50.614093    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:51.033898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:51.033708    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"]
Nov 03 18:16:51.959599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:51.959563    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl" event={"ID":"54fe4529-f34b-4db0-8355-dea8d7ce5aa0","Type":"ContainerStarted","Data":"c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc"}
Nov 03 18:16:51.959599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:51.959602    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl" event={"ID":"54fe4529-f34b-4db0-8355-dea8d7ce5aa0","Type":"ContainerStarted","Data":"c6a7321bd10bc54fdc81ca2ae75ed254b3271131eb6ea75278c4037b2d5f467f"}
Nov 03 18:16:52.082402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:52.082346    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl" podStartSLOduration=2.082328164 podStartE2EDuration="2.082328164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:52.079853839 +0000 UTC m=+2591.539024826" watchObservedRunningTime="2026-11-03 18:16:52.082328164 +0000 UTC m=+2591.541499150"
Nov 03 18:16:52.615070 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:52.615038    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:16:53.037531 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:53.037495    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"]
Nov 03 18:16:53.088337 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:53.088305    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-42c162-t2hcl"]
Nov 03 18:16:54.712665 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:54.712629    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c5412f-fb67-47ac-91b5-bff1a6561a2c" path="/var/lib/kubelet/pods/d7c5412f-fb67-47ac-91b5-bff1a6561a2c/volumes"
Nov 03 18:16:55.391699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:55.391661    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"]
Nov 03 18:16:55.391979 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:16:55.391950    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy" containerID="cri-o://db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783" gracePeriod=30
Nov 03 18:17:00.614832 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.614789    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:17:00.628062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.628036    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:17:00.642390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.642368    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 18:17:00.754151 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754119    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-credential-socket\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754178    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-certs\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754203    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-data\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754245    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-token\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754275    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-socket\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754308    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istiod-ca-cert\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754351    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-envoy\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754418    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-kube-api-access-2pl74\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-kube-api-access-2pl74\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754460    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-podinfo\") pod \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\" (UID: \"f4c7bb05-062a-4e4c-b162-d54009ce9c2b\") "
Nov 03 18:17:00.754730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754489    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-credential-socket" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.754730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754664    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-socket" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.754730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754720    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-certs" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.755013 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.754903    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-data" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.755111 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.755086    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istiod-ca-cert" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:00.755238 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.755216    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.755328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.755244    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.755328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.755258    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.755328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.755271    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.755328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.755284    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.756995 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.756970    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-envoy" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:00.757311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.757293    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-podinfo" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:17:00.757559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.757534    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-token" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:00.757559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.757541    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-kube-api-access-2pl74" pod "f4c7bb05-062a-4e4c-b162-d54009ce9c2b" (UID: "f4c7bb05-062a-4e4c-b162-d54009ce9c2b"). InnerVolumeSpecName "kube-api-access-2pl74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:00.858914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.856773    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.858914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.856807    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.858914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.856824    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pl74\" (UniqueName: \"kubernetes.io/projected/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-kube-api-access-2pl74\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:00.858914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:00.856840    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c7bb05-062a-4e4c-b162-d54009ce9c2b-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:01.015267 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.015191    4493 generic.go:356] "Generic (PLEG): container finished" podID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerID="db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783" exitCode=0
Nov 03 18:17:01.015267 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.015236    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"
Nov 03 18:17:01.015512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.015280    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" event={"ID":"f4c7bb05-062a-4e4c-b162-d54009ce9c2b","Type":"ContainerDied","Data":"db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783"}
Nov 03 18:17:01.015512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.015344    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4" event={"ID":"f4c7bb05-062a-4e4c-b162-d54009ce9c2b","Type":"ContainerDied","Data":"be9a2542aada33bb7d354c9b099bdde1498cf89e97bdc55f35388d3b7359c5d7"}
Nov 03 18:17:01.015512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.015378    4493 scope.go:122] "RemoveContainer" containerID="db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783"
Nov 03 18:17:01.027570 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.027549    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:17:01.028847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.028830    4493 scope.go:122] "RemoveContainer" containerID="db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783"
Nov 03 18:17:01.029157 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:17:01.029137    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783\": container with ID starting with db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783 not found: ID does not exist" containerID="db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783"
Nov 03 18:17:01.029237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.029166    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783"} err="failed to get container status \"db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783\": rpc error: code = NotFound desc = could not find container \"db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783\": container with ID starting with db6db9c78e3a474d2c3e9b9c07c8b7c9ea84066d88bb16875f72f4aff7545783 not found: ID does not exist"
Nov 03 18:17:01.311014 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.310924    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"]
Nov 03 18:17:01.316497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:01.316469    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-dgps4"]
Nov 03 18:17:02.487463 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:02.487425    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"]
Nov 03 18:17:02.487955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:02.487638    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" containerID="cri-o://72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075" gracePeriod=30
Nov 03 18:17:02.713350 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:02.713315    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" path="/var/lib/kubelet/pods/f4c7bb05-062a-4e4c-b162-d54009ce9c2b/volumes"
Nov 03 18:17:03.826838 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:03.826806    4493 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:17:03.827337 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:03.827307    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:17:06.992003 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:17:06.991968    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:17:07.796608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.796581    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 18:17:07.865904 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.865836    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-socket\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.865904 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.865906    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-credential-socket\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.865966    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-certs\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866017    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-envoy\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866052    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-data\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866081    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-socket" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:07.866165 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866145    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istiod-ca-cert\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866210    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-podinfo\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866230    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-credential-socket" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:07.866441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866240    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-kube-api-access-2zt95\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-kube-api-access-2zt95\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866338    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-token\") pod \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\" (UID: \"fbb5b51b-a416-4b17-a69a-9a3f32938a13\") "
Nov 03 18:17:07.866441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866340    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-certs" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:07.866441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866360    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-data" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:07.866769 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866512    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istiod-ca-cert" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:07.867059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866838    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.867059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866885    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.867059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866902    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.867059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.866958    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.867340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.867097    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.868470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.868448    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-token" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:07.868571 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.868488    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-envoy" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:17:07.868646 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.868590    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-podinfo" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:17:07.868699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.868651    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-kube-api-access-2zt95" pod "fbb5b51b-a416-4b17-a69a-9a3f32938a13" (UID: "fbb5b51b-a416-4b17-a69a-9a3f32938a13"). InnerVolumeSpecName "kube-api-access-2zt95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:07.968221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.968191    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.968221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.968216    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zt95\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-kube-api-access-2zt95\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.968221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.968226    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:07.968456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:07.968235    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb5b51b-a416-4b17-a69a-9a3f32938a13-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:08.065906 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.065794    4493 generic.go:356] "Generic (PLEG): container finished" podID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerID="72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075" exitCode=0
Nov 03 18:17:08.065906 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.065866    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"
Nov 03 18:17:08.066362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.065863    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" event={"ID":"fbb5b51b-a416-4b17-a69a-9a3f32938a13","Type":"ContainerDied","Data":"72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075"}
Nov 03 18:17:08.066362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.066024    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr" event={"ID":"fbb5b51b-a416-4b17-a69a-9a3f32938a13","Type":"ContainerDied","Data":"e25fb93dd3ec20b9d846d616344bcc4470664cd37e214cbb661a840e4feba2df"}
Nov 03 18:17:08.066362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.066045    4493 scope.go:122] "RemoveContainer" containerID="72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075"
Nov 03 18:17:08.076783 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.076762    4493 scope.go:122] "RemoveContainer" containerID="72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075"
Nov 03 18:17:08.077115 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:17:08.077094    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075\": container with ID starting with 72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075 not found: ID does not exist" containerID="72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075"
Nov 03 18:17:08.077239 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.077123    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075"} err="failed to get container status \"72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075\": rpc error: code = NotFound desc = could not find container \"72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075\": container with ID starting with 72b0316d761f966217d048c1de45de5ad3e9a8ed0ff77776a65e469f3e247075 not found: ID does not exist"
Nov 03 18:17:08.238520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.238471    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"]
Nov 03 18:17:08.246261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.246223    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dm2sr"]
Nov 03 18:17:08.712174 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:08.712140    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" path="/var/lib/kubelet/pods/fbb5b51b-a416-4b17-a69a-9a3f32938a13/volumes"
Nov 03 18:17:12.286426 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.286379    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"]
Nov 03 18:17:12.287131 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.287109    4493 cpu_manager.go:414] "containerMap: removing container" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy"
Nov 03 18:17:12.287240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.287133    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy"
Nov 03 18:17:12.287240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.287159    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy"
Nov 03 18:17:12.287240 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.287166    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy"
Nov 03 18:17:12.287418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.287279    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c7bb05-062a-4e4c-b162-d54009ce9c2b" containerName="istio-proxy"
Nov 03 18:17:12.287418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.287295    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb5b51b-a416-4b17-a69a-9a3f32938a13" containerName="istio-proxy"
Nov 03 18:17:12.291859 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.291836    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.316383 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.316339    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxbs\" (UniqueName: \"kubernetes.io/projected/0719be4b-6804-42da-ba2f-6f6e3088b4af-kube-api-access-crxbs\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.316556 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.316545    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.316618 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.316600    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.350649 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.350609    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"]
Nov 03 18:17:12.397464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.397427    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:17:12.397645 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.397488    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:17:12.397645 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.397415    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:17:12.397645 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.397568    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:17:12.417853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.417819    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.418050 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.417902    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxbs\" (UniqueName: \"kubernetes.io/projected/0719be4b-6804-42da-ba2f-6f6e3088b4af-kube-api-access-crxbs\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.418050 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.418018    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.418607 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.418590    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.420327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.420309    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.481799 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.481764    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxbs\" (UniqueName: \"kubernetes.io/projected/0719be4b-6804-42da-ba2f-6f6e3088b4af-kube-api-access-crxbs\") pod \"maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.605314 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.605234    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:12.991515 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:12.991484    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"]
Nov 03 18:17:12.992429 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:17:12.992388    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0719be4b_6804_42da_ba2f_6f6e3088b4af.slice/crio-7859347ff8674eb7ad32fe8d7be872e09d3f85993f7218ac6b670e75fd64633c WatchSource:0}: Error finding container 7859347ff8674eb7ad32fe8d7be872e09d3f85993f7218ac6b670e75fd64633c: Status 404 returned error can't find the container with id 7859347ff8674eb7ad32fe8d7be872e09d3f85993f7218ac6b670e75fd64633c
Nov 03 18:17:13.098920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.098879    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr" event={"ID":"0719be4b-6804-42da-ba2f-6f6e3088b4af","Type":"ContainerStarted","Data":"7859347ff8674eb7ad32fe8d7be872e09d3f85993f7218ac6b670e75fd64633c"}
Nov 03 18:17:13.403574 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.403539    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"]
Nov 03 18:17:13.407221 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.407203    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.431386 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.431361    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.431530 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.431507    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6pk\" (UniqueName: \"kubernetes.io/projected/d52ebf0a-23bd-45de-a224-9feed6254685-kube-api-access-kw6pk\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.431623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.431607    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.463269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.463233    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"]
Nov 03 18:17:13.467064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.467029    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.488619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.488580    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"]
Nov 03 18:17:13.527397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.526896    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"]
Nov 03 18:17:13.532916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.532886    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.533085 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.533037    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqff\" (UniqueName: \"kubernetes.io/projected/9a089dcc-953c-4ef4-9889-b945702e8bf8-kube-api-access-qcqff\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.533158 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.533117    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6pk\" (UniqueName: \"kubernetes.io/projected/d52ebf0a-23bd-45de-a224-9feed6254685-kube-api-access-kw6pk\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.533216 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.533182    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.533295 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.533279    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.533386 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.533361    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.534038 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.534017    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.535770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.535751    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.596706 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.596636    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6pk\" (UniqueName: \"kubernetes.io/projected/d52ebf0a-23bd-45de-a224-9feed6254685-kube-api-access-kw6pk\") pod \"maas-api-revoke-sub-6c9b5fad-vdmjx\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.634353 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.634316    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqff\" (UniqueName: \"kubernetes.io/projected/9a089dcc-953c-4ef4-9889-b945702e8bf8-kube-api-access-qcqff\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.634528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.634373    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.634528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.634476    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.635037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.635015    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.636829 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.636806    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.698348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.698316    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqff\" (UniqueName: \"kubernetes.io/projected/9a089dcc-953c-4ef4-9889-b945702e8bf8-kube-api-access-qcqff\") pod \"maas-api-revoke-sub-8d079df1-rqb85\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:13.719617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.719586    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:13.782885 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:13.782410    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:14.101917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:14.101884    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"]
Nov 03 18:17:14.102374 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:17:14.102350    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52ebf0a_23bd_45de_a224_9feed6254685.slice/crio-8a8c2534670c356e81081d0ff1e06f7eda4b7ef8d0f672bda5edb35c7fcc0c59 WatchSource:0}: Error finding container 8a8c2534670c356e81081d0ff1e06f7eda4b7ef8d0f672bda5edb35c7fcc0c59: Status 404 returned error can't find the container with id 8a8c2534670c356e81081d0ff1e06f7eda4b7ef8d0f672bda5edb35c7fcc0c59
Nov 03 18:17:14.108218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:14.108188    4493 generic.go:356] "Generic (PLEG): container finished" podID="0719be4b-6804-42da-ba2f-6f6e3088b4af" containerID="02ede4170ef46fe3275990f18b2537a90f7e40cbe28174b24389ead1cf749370" exitCode=0
Nov 03 18:17:14.108319 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:14.108277    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr" event={"ID":"0719be4b-6804-42da-ba2f-6f6e3088b4af","Type":"ContainerDied","Data":"02ede4170ef46fe3275990f18b2537a90f7e40cbe28174b24389ead1cf749370"}
Nov 03 18:17:14.215429 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:14.215399    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"]
Nov 03 18:17:14.217225 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:17:14.217200    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a089dcc_953c_4ef4_9889_b945702e8bf8.slice/crio-7bdada7368e8592c67af762e503c96d93575d2787b9acac54d914f942c7b8d58 WatchSource:0}: Error finding container 7bdada7368e8592c67af762e503c96d93575d2787b9acac54d914f942c7b8d58: Status 404 returned error can't find the container with id 7bdada7368e8592c67af762e503c96d93575d2787b9acac54d914f942c7b8d58
Nov 03 18:17:15.117108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.116923    4493 generic.go:356] "Generic (PLEG): container finished" podID="d52ebf0a-23bd-45de-a224-9feed6254685" containerID="8ff9ba2e75cb2b940c8326407d418e8e28242e75331a21b4a5d7e084a61b706b" exitCode=0
Nov 03 18:17:15.117108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.116996    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx" event={"ID":"d52ebf0a-23bd-45de-a224-9feed6254685","Type":"ContainerDied","Data":"8ff9ba2e75cb2b940c8326407d418e8e28242e75331a21b4a5d7e084a61b706b"}
Nov 03 18:17:15.117108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.117069    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx" event={"ID":"d52ebf0a-23bd-45de-a224-9feed6254685","Type":"ContainerStarted","Data":"8a8c2534670c356e81081d0ff1e06f7eda4b7ef8d0f672bda5edb35c7fcc0c59"}
Nov 03 18:17:15.119325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.119296    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85" event={"ID":"9a089dcc-953c-4ef4-9889-b945702e8bf8","Type":"ContainerStarted","Data":"7dc702d6b09fc9e36ccb0c4f5c0775ac3375f92ebe2a80d1a3b2815da3c1f6e7"}
Nov 03 18:17:15.119470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.119353    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85" event={"ID":"9a089dcc-953c-4ef4-9889-b945702e8bf8","Type":"ContainerStarted","Data":"7bdada7368e8592c67af762e503c96d93575d2787b9acac54d914f942c7b8d58"}
Nov 03 18:17:15.270017 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.269991    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:15.355597 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.355561    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-xks-ca\") pod \"0719be4b-6804-42da-ba2f-6f6e3088b4af\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") "
Nov 03 18:17:15.355774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.355625    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-openshift-ca\") pod \"0719be4b-6804-42da-ba2f-6f6e3088b4af\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") "
Nov 03 18:17:15.355774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.355732    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0719be4b-6804-42da-ba2f-6f6e3088b4af-kube-api-access-crxbs\" (UniqueName: \"kubernetes.io/projected/0719be4b-6804-42da-ba2f-6f6e3088b4af-kube-api-access-crxbs\") pod \"0719be4b-6804-42da-ba2f-6f6e3088b4af\" (UID: \"0719be4b-6804-42da-ba2f-6f6e3088b4af\") "
Nov 03 18:17:15.356097 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.356068    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-openshift-ca" pod "0719be4b-6804-42da-ba2f-6f6e3088b4af" (UID: "0719be4b-6804-42da-ba2f-6f6e3088b4af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:15.357886 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.357851    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-xks-ca" pod "0719be4b-6804-42da-ba2f-6f6e3088b4af" (UID: "0719be4b-6804-42da-ba2f-6f6e3088b4af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:15.357963 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.357943    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0719be4b-6804-42da-ba2f-6f6e3088b4af-kube-api-access-crxbs" pod "0719be4b-6804-42da-ba2f-6f6e3088b4af" (UID: "0719be4b-6804-42da-ba2f-6f6e3088b4af"). InnerVolumeSpecName "kube-api-access-crxbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:15.361367 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.361320    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85" podStartSLOduration=2.36130864 podStartE2EDuration="2.36130864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:15.359236035 +0000 UTC m=+2614.818407021" watchObservedRunningTime="2026-11-03 18:17:15.36130864 +0000 UTC m=+2614.820479626"
Nov 03 18:17:15.456993 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.456958    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:15.456993 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.456988    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0719be4b-6804-42da-ba2f-6f6e3088b4af-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:15.456993 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:15.456998    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxbs\" (UniqueName: \"kubernetes.io/projected/0719be4b-6804-42da-ba2f-6f6e3088b4af-kube-api-access-crxbs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:16.126268 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.126242    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"
Nov 03 18:17:16.126760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.126238    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr" event={"ID":"0719be4b-6804-42da-ba2f-6f6e3088b4af","Type":"ContainerDied","Data":"7859347ff8674eb7ad32fe8d7be872e09d3f85993f7218ac6b670e75fd64633c"}
Nov 03 18:17:16.126760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.126401    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7859347ff8674eb7ad32fe8d7be872e09d3f85993f7218ac6b670e75fd64633c"
Nov 03 18:17:16.127970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.127944    4493 generic.go:356] "Generic (PLEG): container finished" podID="9a089dcc-953c-4ef4-9889-b945702e8bf8" containerID="7dc702d6b09fc9e36ccb0c4f5c0775ac3375f92ebe2a80d1a3b2815da3c1f6e7" exitCode=0
Nov 03 18:17:16.128082 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.127992    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85" event={"ID":"9a089dcc-953c-4ef4-9889-b945702e8bf8","Type":"ContainerDied","Data":"7dc702d6b09fc9e36ccb0c4f5c0775ac3375f92ebe2a80d1a3b2815da3c1f6e7"}
Nov 03 18:17:16.277660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.277630    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:16.366442 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.366408    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-xks-ca\") pod \"d52ebf0a-23bd-45de-a224-9feed6254685\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") "
Nov 03 18:17:16.366587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.366466    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-openshift-ca\") pod \"d52ebf0a-23bd-45de-a224-9feed6254685\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") "
Nov 03 18:17:16.366645 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.366598    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52ebf0a-23bd-45de-a224-9feed6254685-kube-api-access-kw6pk\" (UniqueName: \"kubernetes.io/projected/d52ebf0a-23bd-45de-a224-9feed6254685-kube-api-access-kw6pk\") pod \"d52ebf0a-23bd-45de-a224-9feed6254685\" (UID: \"d52ebf0a-23bd-45de-a224-9feed6254685\") "
Nov 03 18:17:16.366828 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.366803    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-openshift-ca" pod "d52ebf0a-23bd-45de-a224-9feed6254685" (UID: "d52ebf0a-23bd-45de-a224-9feed6254685"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:16.367019 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.367000    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:16.368633 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.368599    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-xks-ca" pod "d52ebf0a-23bd-45de-a224-9feed6254685" (UID: "d52ebf0a-23bd-45de-a224-9feed6254685"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:16.368730 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.368685    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52ebf0a-23bd-45de-a224-9feed6254685-kube-api-access-kw6pk" pod "d52ebf0a-23bd-45de-a224-9feed6254685" (UID: "d52ebf0a-23bd-45de-a224-9feed6254685"). InnerVolumeSpecName "kube-api-access-kw6pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:16.468248 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.468214    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw6pk\" (UniqueName: \"kubernetes.io/projected/d52ebf0a-23bd-45de-a224-9feed6254685-kube-api-access-kw6pk\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:16.468248 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:16.468243    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52ebf0a-23bd-45de-a224-9feed6254685-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:17.134577 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.134490    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"
Nov 03 18:17:17.134577 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.134543    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx" event={"ID":"d52ebf0a-23bd-45de-a224-9feed6254685","Type":"ContainerDied","Data":"8a8c2534670c356e81081d0ff1e06f7eda4b7ef8d0f672bda5edb35c7fcc0c59"}
Nov 03 18:17:17.135030 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.134599    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a8c2534670c356e81081d0ff1e06f7eda4b7ef8d0f672bda5edb35c7fcc0c59"
Nov 03 18:17:17.277092 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.277065    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:17.379149 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.379112    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-xks-ca\") pod \"9a089dcc-953c-4ef4-9889-b945702e8bf8\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") "
Nov 03 18:17:17.379347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.379168    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a089dcc-953c-4ef4-9889-b945702e8bf8-kube-api-access-qcqff\" (UniqueName: \"kubernetes.io/projected/9a089dcc-953c-4ef4-9889-b945702e8bf8-kube-api-access-qcqff\") pod \"9a089dcc-953c-4ef4-9889-b945702e8bf8\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") "
Nov 03 18:17:17.379347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.379336    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-openshift-ca\") pod \"9a089dcc-953c-4ef4-9889-b945702e8bf8\" (UID: \"9a089dcc-953c-4ef4-9889-b945702e8bf8\") "
Nov 03 18:17:17.379700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.379676    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-openshift-ca" pod "9a089dcc-953c-4ef4-9889-b945702e8bf8" (UID: "9a089dcc-953c-4ef4-9889-b945702e8bf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:17.379854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.379788    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:17.381460 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.381438    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-xks-ca" pod "9a089dcc-953c-4ef4-9889-b945702e8bf8" (UID: "9a089dcc-953c-4ef4-9889-b945702e8bf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:17.381553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.381504    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a089dcc-953c-4ef4-9889-b945702e8bf8-kube-api-access-qcqff" pod "9a089dcc-953c-4ef4-9889-b945702e8bf8" (UID: "9a089dcc-953c-4ef4-9889-b945702e8bf8"). InnerVolumeSpecName "kube-api-access-qcqff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:17.480673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.480635    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a089dcc-953c-4ef4-9889-b945702e8bf8-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:17.480673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:17.480674    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqff\" (UniqueName: \"kubernetes.io/projected/9a089dcc-953c-4ef4-9889-b945702e8bf8-kube-api-access-qcqff\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:18.143156 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:18.143128    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"
Nov 03 18:17:18.143551 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:18.143124    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85" event={"ID":"9a089dcc-953c-4ef4-9889-b945702e8bf8","Type":"ContainerDied","Data":"7bdada7368e8592c67af762e503c96d93575d2787b9acac54d914f942c7b8d58"}
Nov 03 18:17:18.143551 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:18.143252    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bdada7368e8592c67af762e503c96d93575d2787b9acac54d914f942c7b8d58"
Nov 03 18:17:25.221857 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.221821    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"]
Nov 03 18:17:25.222337 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222322    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0719be4b-6804-42da-ba2f-6f6e3088b4af" containerName="revoke-keys"
Nov 03 18:17:25.222385 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222338    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0719be4b-6804-42da-ba2f-6f6e3088b4af" containerName="revoke-keys"
Nov 03 18:17:25.222385 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222382    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d52ebf0a-23bd-45de-a224-9feed6254685" containerName="revoke-keys"
Nov 03 18:17:25.222453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222388    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52ebf0a-23bd-45de-a224-9feed6254685" containerName="revoke-keys"
Nov 03 18:17:25.222453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222402    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9a089dcc-953c-4ef4-9889-b945702e8bf8" containerName="revoke-keys"
Nov 03 18:17:25.222453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222409    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a089dcc-953c-4ef4-9889-b945702e8bf8" containerName="revoke-keys"
Nov 03 18:17:25.222542 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222482    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a089dcc-953c-4ef4-9889-b945702e8bf8" containerName="revoke-keys"
Nov 03 18:17:25.222542 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222493    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0719be4b-6804-42da-ba2f-6f6e3088b4af" containerName="revoke-keys"
Nov 03 18:17:25.222542 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.222504    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52ebf0a-23bd-45de-a224-9feed6254685" containerName="revoke-keys"
Nov 03 18:17:25.225923 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.225903    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.286258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.286220    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"]
Nov 03 18:17:25.332902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.332846    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0adf17\"/\"kube-root-ca.crt\""
Nov 03 18:17:25.333091 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.332958    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0adf17\"/\"openshift-service-ca.crt\""
Nov 03 18:17:25.333091 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.332990    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0adf17\"/\"default-dockercfg-tq76t\""
Nov 03 18:17:25.333343 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.333319    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0adf17\"/\"test-model-0adf17-kserve-self-signed-certs\""
Nov 03 18:17:25.388670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.385969    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-home\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.389078 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.389049    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-tmp-dir\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.389229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.389095    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-model-cache\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.389229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.389130    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thf7\" (UniqueName: \"kubernetes.io/projected/829de85e-943e-4d8a-a2c4-d517bad23734-kube-api-access-5thf7\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.389229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.389203    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-dshm\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.389422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.389264    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829de85e-943e-4d8a-a2c4-d517bad23734-tls-certs\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491018 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.490907    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-dshm\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491018 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.490974    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829de85e-943e-4d8a-a2c4-d517bad23734-tls-certs\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.491056    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-home\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.491110    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-tmp-dir\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.491131    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-model-cache\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.491153    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thf7\" (UniqueName: \"kubernetes.io/projected/829de85e-943e-4d8a-a2c4-d517bad23734-kube-api-access-5thf7\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.491471    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-home\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.491491    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-tmp-dir\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.491605 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.491535    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-model-cache\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.493567 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.493540    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-dshm\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.493737 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.493709    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829de85e-943e-4d8a-a2c4-d517bad23734-tls-certs\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.554801 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.554766    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thf7\" (UniqueName: \"kubernetes.io/projected/829de85e-943e-4d8a-a2c4-d517bad23734-kube-api-access-5thf7\") pod \"test-model-0adf17-kserve-6fd969dddf-t4prp\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") " pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:25.838584 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:25.838499    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:26.225691 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:17:26.225149    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod829de85e_943e_4d8a_a2c4_d517bad23734.slice/crio-df4a9d1114c4c0978244eb47b22427f87c58132168a3101979579b495418941f WatchSource:0}: Error finding container df4a9d1114c4c0978244eb47b22427f87c58132168a3101979579b495418941f: Status 404 returned error can't find the container with id df4a9d1114c4c0978244eb47b22427f87c58132168a3101979579b495418941f
Nov 03 18:17:26.225691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:26.225311    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"]
Nov 03 18:17:27.201841 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:27.201807    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp" event={"ID":"829de85e-943e-4d8a-a2c4-d517bad23734","Type":"ContainerStarted","Data":"40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d"}
Nov 03 18:17:27.201841 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:27.201847    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp" event={"ID":"829de85e-943e-4d8a-a2c4-d517bad23734","Type":"ContainerStarted","Data":"df4a9d1114c4c0978244eb47b22427f87c58132168a3101979579b495418941f"}
Nov 03 18:17:27.324855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:27.324797    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp" podStartSLOduration=2.324782352 podStartE2EDuration="2.324782352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:27.322457728 +0000 UTC m=+2626.781628713" watchObservedRunningTime="2026-11-03 18:17:27.324782352 +0000 UTC m=+2626.783953337"
Nov 03 18:17:27.839235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:27.839201    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:35.838784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:35.838752    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:35.856193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:35.856169    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:36.269408 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:36.269372    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:17:36.992455 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:17:36.992419    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:17:42.907857 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:42.907824    4493 scope.go:122] "RemoveContainer" containerID="c3136a986b04adb8e3090e4408c6345793aafb8b079b507acb4e274c2283b711"
Nov 03 18:17:42.921507 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:42.921484    4493 scope.go:122] "RemoveContainer" containerID="f1126918de3508dc3faaeb9fe67f49de6e6f46d266538537d0e44910f6c1173d"
Nov 03 18:17:42.930103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:42.930085    4493 scope.go:122] "RemoveContainer" containerID="5174b0a945e1da60e6ee96a05a0375a8a120b38a748da5055ad036d76fc01856"
Nov 03 18:17:42.938154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:42.938136    4493 scope.go:122] "RemoveContainer" containerID="c806696af4ed4965dab79ddb7e8fc15d184ab85c25d1fa85f7cc84e34db0e25f"
Nov 03 18:17:42.946262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:42.946243    4493 scope.go:122] "RemoveContainer" containerID="ddf0fdd09d7e27a41c572530abae23cfaf66843c2dd1cd16129d215aeca04ca7"
Nov 03 18:17:42.954585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:42.954566    4493 scope.go:122] "RemoveContainer" containerID="fc09b5ef239291a9ce7fee5c40f4111168cb27d99f81dd8182d37197938c6d37"
Nov 03 18:17:53.176195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.176118    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"]
Nov 03 18:17:53.181042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.181023    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.237640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.237607    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"]
Nov 03 18:17:53.284634 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.284600    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:17:53.284634 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.284615    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:17:53.284852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.284656    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:17:53.284852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.284677    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:17:53.309607 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.309581    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxn9\" (UniqueName: \"kubernetes.io/projected/14999464-14c6-4094-9167-dfc71d615aad-kube-api-access-mnxn9\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.309771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.309639    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14999464-14c6-4094-9167-dfc71d615aad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.309771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.309715    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14999464-14c6-4094-9167-dfc71d615aad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.411243 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.411210    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxn9\" (UniqueName: \"kubernetes.io/projected/14999464-14c6-4094-9167-dfc71d615aad-kube-api-access-mnxn9\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.411460 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.411276    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14999464-14c6-4094-9167-dfc71d615aad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.411460 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.411321    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14999464-14c6-4094-9167-dfc71d615aad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.412162 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.412135    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14999464-14c6-4094-9167-dfc71d615aad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.413983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.413958    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14999464-14c6-4094-9167-dfc71d615aad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.474648 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.474618    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxn9\" (UniqueName: \"kubernetes.io/projected/14999464-14c6-4094-9167-dfc71d615aad-kube-api-access-mnxn9\") pod \"maas-api-revoke-keys-e2e-api-a-43bdc0-m479s\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.494374 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.494352    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:53.875037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:53.875012    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"]
Nov 03 18:17:53.881632 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:17:53.881149    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14999464_14c6_4094_9167_dfc71d615aad.slice/crio-b04187d5932c2e2599b9815f40637372fd8675883c4c8b06e548cb2423d00455 WatchSource:0}: Error finding container b04187d5932c2e2599b9815f40637372fd8675883c4c8b06e548cb2423d00455: Status 404 returned error can't find the container with id b04187d5932c2e2599b9815f40637372fd8675883c4c8b06e548cb2423d00455
Nov 03 18:17:54.367398 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:54.367362    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s" event={"ID":"14999464-14c6-4094-9167-dfc71d615aad","Type":"ContainerStarted","Data":"bc22a51dc9cd908b23312b6fc45bd02850a3f8cc25985f44a7fb27bd1724d8e3"}
Nov 03 18:17:54.367855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:54.367416    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s" event={"ID":"14999464-14c6-4094-9167-dfc71d615aad","Type":"ContainerStarted","Data":"b04187d5932c2e2599b9815f40637372fd8675883c4c8b06e548cb2423d00455"}
Nov 03 18:17:54.489368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:54.489312    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s" podStartSLOduration=1.489296261 podStartE2EDuration="1.489296261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:17:54.487050157 +0000 UTC m=+2653.946221142" watchObservedRunningTime="2026-11-03 18:17:54.489296261 +0000 UTC m=+2653.948467246"
Nov 03 18:17:55.375528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:55.375498    4493 generic.go:356] "Generic (PLEG): container finished" podID="14999464-14c6-4094-9167-dfc71d615aad" containerID="bc22a51dc9cd908b23312b6fc45bd02850a3f8cc25985f44a7fb27bd1724d8e3" exitCode=0
Nov 03 18:17:55.375949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:55.375550    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s" event={"ID":"14999464-14c6-4094-9167-dfc71d615aad","Type":"ContainerDied","Data":"bc22a51dc9cd908b23312b6fc45bd02850a3f8cc25985f44a7fb27bd1724d8e3"}
Nov 03 18:17:56.531244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.531211    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:56.649093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.649009    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14999464-14c6-4094-9167-dfc71d615aad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14999464-14c6-4094-9167-dfc71d615aad-maas-api-xks-ca\") pod \"14999464-14c6-4094-9167-dfc71d615aad\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") "
Nov 03 18:17:56.649093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.649073    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14999464-14c6-4094-9167-dfc71d615aad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14999464-14c6-4094-9167-dfc71d615aad-maas-api-openshift-ca\") pod \"14999464-14c6-4094-9167-dfc71d615aad\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") "
Nov 03 18:17:56.649321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.649148    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14999464-14c6-4094-9167-dfc71d615aad-kube-api-access-mnxn9\" (UniqueName: \"kubernetes.io/projected/14999464-14c6-4094-9167-dfc71d615aad-kube-api-access-mnxn9\") pod \"14999464-14c6-4094-9167-dfc71d615aad\" (UID: \"14999464-14c6-4094-9167-dfc71d615aad\") "
Nov 03 18:17:56.649432 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.649409    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14999464-14c6-4094-9167-dfc71d615aad-maas-api-openshift-ca" pod "14999464-14c6-4094-9167-dfc71d615aad" (UID: "14999464-14c6-4094-9167-dfc71d615aad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:17:56.651323 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.651289    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14999464-14c6-4094-9167-dfc71d615aad-kube-api-access-mnxn9" pod "14999464-14c6-4094-9167-dfc71d615aad" (UID: "14999464-14c6-4094-9167-dfc71d615aad"). InnerVolumeSpecName "kube-api-access-mnxn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:17:56.651481 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.651341    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14999464-14c6-4094-9167-dfc71d615aad-maas-api-xks-ca" pod "14999464-14c6-4094-9167-dfc71d615aad" (UID: "14999464-14c6-4094-9167-dfc71d615aad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:17:56.750744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.750705    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnxn9\" (UniqueName: \"kubernetes.io/projected/14999464-14c6-4094-9167-dfc71d615aad-kube-api-access-mnxn9\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:56.750744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.750738    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14999464-14c6-4094-9167-dfc71d615aad-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:56.750744 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:56.750747    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14999464-14c6-4094-9167-dfc71d615aad-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:17:57.388797 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:57.388763    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s" event={"ID":"14999464-14c6-4094-9167-dfc71d615aad","Type":"ContainerDied","Data":"b04187d5932c2e2599b9815f40637372fd8675883c4c8b06e548cb2423d00455"}
Nov 03 18:17:57.388797 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:57.388800    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"
Nov 03 18:17:57.389167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:17:57.388814    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04187d5932c2e2599b9815f40637372fd8675883c4c8b06e548cb2423d00455"
Nov 03 18:18:02.796217 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:02.796177    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"]
Nov 03 18:18:02.846814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:02.846777    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-c76280-vzth6"]
Nov 03 18:18:04.712461 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:04.712425    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3294a2-6cba-4547-b528-0cf702991cdf" path="/var/lib/kubelet/pods/dd3294a2-6cba-4547-b528-0cf702991cdf/volumes"
Nov 03 18:18:06.992697 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:18:06.992646    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:18:11.028629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:11.028596    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"]
Nov 03 18:18:11.029083 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:11.028799    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy" containerID="cri-o://ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7" gracePeriod=30
Nov 03 18:18:16.285721 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.285691    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 18:18:16.371763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.371667    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-credential-socket\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.371763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.371718    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-envoy\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.371763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.371749    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded41810-cf11-46b9-8c36-f3a7f56647ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded41810-cf11-46b9-8c36-f3a7f56647ef-istiod-ca-cert\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.372098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.371785    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-certs\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.372098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.371898    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-kube-api-access-dzpdx\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-kube-api-access-dzpdx\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.372098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.371928    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-podinfo\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.372098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372048    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-data\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.372098 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372062    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-credential-socket" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:16.372384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372101    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-socket\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.372384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372175    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-token\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-token\") pod \"ded41810-cf11-46b9-8c36-f3a7f56647ef\" (UID: \"ded41810-cf11-46b9-8c36-f3a7f56647ef\") "
Nov 03 18:18:16.372384 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372250    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-certs" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:16.372544 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372379    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-data" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:16.372544 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372323    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded41810-cf11-46b9-8c36-f3a7f56647ef-istiod-ca-cert" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:16.372818 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372619    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-socket" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:16.372928 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372905    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.373002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372934    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.373002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372951    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.373002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372968    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded41810-cf11-46b9-8c36-f3a7f56647ef-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.373002 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.372982    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.374325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.374298    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-podinfo" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:18:16.374441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.374327    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-envoy" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:18:16.374441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.374387    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-kube-api-access-dzpdx" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "kube-api-access-dzpdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:16.374768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.374749    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-token" pod "ded41810-cf11-46b9-8c36-f3a7f56647ef" (UID: "ded41810-cf11-46b9-8c36-f3a7f56647ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:16.474677 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.474643    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.474677 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.474676    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.474932 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.474690    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzpdx\" (UniqueName: \"kubernetes.io/projected/ded41810-cf11-46b9-8c36-f3a7f56647ef-kube-api-access-dzpdx\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.474932 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.474704    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded41810-cf11-46b9-8c36-f3a7f56647ef-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:16.499285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.499241    4493 generic.go:356] "Generic (PLEG): container finished" podID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerID="ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7" exitCode=0
Nov 03 18:18:16.499501 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.499300    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"
Nov 03 18:18:16.499501 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.499340    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" event={"ID":"ded41810-cf11-46b9-8c36-f3a7f56647ef","Type":"ContainerDied","Data":"ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7"}
Nov 03 18:18:16.499501 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.499418    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488" event={"ID":"ded41810-cf11-46b9-8c36-f3a7f56647ef","Type":"ContainerDied","Data":"b5ffc95612ab6569df23bfce74c49c8c083a016150c698df748fe731ae14b9a9"}
Nov 03 18:18:16.499501 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.499454    4493 scope.go:122] "RemoveContainer" containerID="ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7"
Nov 03 18:18:16.510905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.510883    4493 scope.go:122] "RemoveContainer" containerID="ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7"
Nov 03 18:18:16.511206 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:18:16.511186    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7\": container with ID starting with ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7 not found: ID does not exist" containerID="ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7"
Nov 03 18:18:16.511278 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.511215    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7"} err="failed to get container status \"ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7\": rpc error: code = NotFound desc = could not find container \"ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7\": container with ID starting with ce381a543970a266fe66495b6f60aabcc9636d15d3dedb1ee37d82e983e0f4b7 not found: ID does not exist"
Nov 03 18:18:16.671086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.671006    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"]
Nov 03 18:18:16.676996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.676964    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-g7488"]
Nov 03 18:18:16.711993 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:16.711956    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" path="/var/lib/kubelet/pods/ded41810-cf11-46b9-8c36-f3a7f56647ef/volumes"
Nov 03 18:18:29.492029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:29.491990    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"]
Nov 03 18:18:29.542436 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:29.542405    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bd8cf-sxglr"]
Nov 03 18:18:30.711953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:30.711915    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0719be4b-6804-42da-ba2f-6f6e3088b4af" path="/var/lib/kubelet/pods/0719be4b-6804-42da-ba2f-6f6e3088b4af/volumes"
Nov 03 18:18:32.061763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:32.061726    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"]
Nov 03 18:18:32.112597 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:32.112555    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6c866d2-qbkts"]
Nov 03 18:18:32.115106 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:32.115081    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"]
Nov 03 18:18:32.117214 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:32.117190    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a03c71e-m6zgx"]
Nov 03 18:18:32.712409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:32.712376    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cace8e1-9202-4eca-a19f-493ed9705514" path="/var/lib/kubelet/pods/8cace8e1-9202-4eca-a19f-493ed9705514/volumes"
Nov 03 18:18:32.712917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:32.712896    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efb4c96-865e-4dd6-a7c1-816911f15d76" path="/var/lib/kubelet/pods/8efb4c96-865e-4dd6-a7c1-816911f15d76/volumes"
Nov 03 18:18:36.994165 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:18:36.994126    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:18:38.208640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.208611    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"]
Nov 03 18:18:38.209080 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.209066    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy"
Nov 03 18:18:38.209126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.209082    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy"
Nov 03 18:18:38.209126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.209119    4493 cpu_manager.go:414] "containerMap: removing container" podUID="14999464-14c6-4094-9167-dfc71d615aad" containerName="revoke-keys"
Nov 03 18:18:38.209193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.209127    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14999464-14c6-4094-9167-dfc71d615aad" containerName="revoke-keys"
Nov 03 18:18:38.209232 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.209195    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="14999464-14c6-4094-9167-dfc71d615aad" containerName="revoke-keys"
Nov 03 18:18:38.209232 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.209213    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded41810-cf11-46b9-8c36-f3a7f56647ef" containerName="istio-proxy"
Nov 03 18:18:38.212391 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.212371    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.274452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.274427    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"]
Nov 03 18:18:38.304265 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.304235    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:18:38.304417 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.304315    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:18:38.304417 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.304338    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:18:38.313825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.313803    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bf6t\" (UniqueName: \"kubernetes.io/projected/73d8ef9a-b646-4705-adfa-b4ea586eb681-kube-api-access-8bf6t\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.313967 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.313846    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.314015 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.313993    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.319427 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.319407    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:18:38.415246 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.415212    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.415381 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.415296    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bf6t\" (UniqueName: \"kubernetes.io/projected/73d8ef9a-b646-4705-adfa-b4ea586eb681-kube-api-access-8bf6t\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.415381 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.415332    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.415845 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.415825    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.417776 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.417751    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.480132 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.480107    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bf6t\" (UniqueName: \"kubernetes.io/projected/73d8ef9a-b646-4705-adfa-b4ea586eb681-kube-api-access-8bf6t\") pod \"maas-api-revoke-keys-e2e-api-b-44769b-grfrp\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.525102 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.525076    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:38.904460 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:38.904432    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"]
Nov 03 18:18:38.906024 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:18:38.905995    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d8ef9a_b646_4705_adfa_b4ea586eb681.slice/crio-0ee0cc881c8255125234ebe1f5873527fbd74eeb19f6f2a82f755de838a55aed WatchSource:0}: Error finding container 0ee0cc881c8255125234ebe1f5873527fbd74eeb19f6f2a82f755de838a55aed: Status 404 returned error can't find the container with id 0ee0cc881c8255125234ebe1f5873527fbd74eeb19f6f2a82f755de838a55aed
Nov 03 18:18:39.638290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:39.638217    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp" event={"ID":"73d8ef9a-b646-4705-adfa-b4ea586eb681","Type":"ContainerStarted","Data":"18f77614a35c2f3a3af7d59e6b68829c5e5fa96de8306c3981cc1bb9469719aa"}
Nov 03 18:18:39.638290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:39.638267    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp" event={"ID":"73d8ef9a-b646-4705-adfa-b4ea586eb681","Type":"ContainerStarted","Data":"0ee0cc881c8255125234ebe1f5873527fbd74eeb19f6f2a82f755de838a55aed"}
Nov 03 18:18:39.760144 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:39.760091    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp" podStartSLOduration=1.760076464 podStartE2EDuration="1.760076464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:18:39.758525326 +0000 UTC m=+2699.217696312" watchObservedRunningTime="2026-11-03 18:18:39.760076464 +0000 UTC m=+2699.219247450"
Nov 03 18:18:40.645766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:40.645731    4493 generic.go:356] "Generic (PLEG): container finished" podID="73d8ef9a-b646-4705-adfa-b4ea586eb681" containerID="18f77614a35c2f3a3af7d59e6b68829c5e5fa96de8306c3981cc1bb9469719aa" exitCode=0
Nov 03 18:18:40.645766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:40.645768    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp" event={"ID":"73d8ef9a-b646-4705-adfa-b4ea586eb681","Type":"ContainerDied","Data":"18f77614a35c2f3a3af7d59e6b68829c5e5fa96de8306c3981cc1bb9469719aa"}
Nov 03 18:18:41.796327 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.796298    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:41.853278 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.853248    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-openshift-ca\") pod \"73d8ef9a-b646-4705-adfa-b4ea586eb681\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") "
Nov 03 18:18:41.853443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.853415    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d8ef9a-b646-4705-adfa-b4ea586eb681-kube-api-access-8bf6t\" (UniqueName: \"kubernetes.io/projected/73d8ef9a-b646-4705-adfa-b4ea586eb681-kube-api-access-8bf6t\") pod \"73d8ef9a-b646-4705-adfa-b4ea586eb681\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") "
Nov 03 18:18:41.853495 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.853469    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-xks-ca\") pod \"73d8ef9a-b646-4705-adfa-b4ea586eb681\" (UID: \"73d8ef9a-b646-4705-adfa-b4ea586eb681\") "
Nov 03 18:18:41.853654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.853634    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-openshift-ca" pod "73d8ef9a-b646-4705-adfa-b4ea586eb681" (UID: "73d8ef9a-b646-4705-adfa-b4ea586eb681"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:18:41.853862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.853848    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:41.855761 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.855737    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-xks-ca" pod "73d8ef9a-b646-4705-adfa-b4ea586eb681" (UID: "73d8ef9a-b646-4705-adfa-b4ea586eb681"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:18:41.855897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.855761    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d8ef9a-b646-4705-adfa-b4ea586eb681-kube-api-access-8bf6t" pod "73d8ef9a-b646-4705-adfa-b4ea586eb681" (UID: "73d8ef9a-b646-4705-adfa-b4ea586eb681"). InnerVolumeSpecName "kube-api-access-8bf6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:18:41.954681 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.954606    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bf6t\" (UniqueName: \"kubernetes.io/projected/73d8ef9a-b646-4705-adfa-b4ea586eb681-kube-api-access-8bf6t\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:41.954681 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:41.954636    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d8ef9a-b646-4705-adfa-b4ea586eb681-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:18:42.660335 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:42.660298    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp" event={"ID":"73d8ef9a-b646-4705-adfa-b4ea586eb681","Type":"ContainerDied","Data":"0ee0cc881c8255125234ebe1f5873527fbd74eeb19f6f2a82f755de838a55aed"}
Nov 03 18:18:42.660509 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:42.660352    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee0cc881c8255125234ebe1f5873527fbd74eeb19f6f2a82f755de838a55aed"
Nov 03 18:18:42.660509 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:42.660314    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"
Nov 03 18:18:43.004733 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:43.004702    4493 scope.go:122] "RemoveContainer" containerID="4775661a59250b3a5bbdbfdbb1c4fb8be0d59cddfc00119c0caa620adc21f359"
Nov 03 18:18:43.014243 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:18:43.014225    4493 scope.go:122] "RemoveContainer" containerID="d330059af3a2a5caeb1880501817c1b77db0149632263a20a3594c4408352515"
Nov 03 18:19:06.991927 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:19:06.991891    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:19:11.863363 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:11.863329    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"]
Nov 03 18:19:11.913656 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:11.913622    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-43bdc0-m479s"]
Nov 03 18:19:12.711931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:12.711896    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14999464-14c6-4094-9167-dfc71d615aad" path="/var/lib/kubelet/pods/14999464-14c6-4094-9167-dfc71d615aad/volumes"
Nov 03 18:19:21.667448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.667373    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"]
Nov 03 18:19:21.667853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.667840    4493 cpu_manager.go:414] "containerMap: removing container" podUID="73d8ef9a-b646-4705-adfa-b4ea586eb681" containerName="revoke-keys"
Nov 03 18:19:21.667983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.667854    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d8ef9a-b646-4705-adfa-b4ea586eb681" containerName="revoke-keys"
Nov 03 18:19:21.667983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.667972    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d8ef9a-b646-4705-adfa-b4ea586eb681" containerName="revoke-keys"
Nov 03 18:19:21.671093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.671077    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.731086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.731059    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"]
Nov 03 18:19:21.777525 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.777496    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:19:21.777708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.777529    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:19:21.777708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.777556    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:19:21.777708 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.777661    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:19:21.799471 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.799447    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.799614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.799500    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.799614 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.799526    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmslq\" (UniqueName: \"kubernetes.io/projected/1c141b70-5fe6-4ae8-a425-23208a33537b-kube-api-access-wmslq\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.901220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.901192    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.901418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.901268    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.901418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.901309    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmslq\" (UniqueName: \"kubernetes.io/projected/1c141b70-5fe6-4ae8-a425-23208a33537b-kube-api-access-wmslq\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.901932 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.901907    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.903771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.903750    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.964755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.964725    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmslq\" (UniqueName: \"kubernetes.io/projected/1c141b70-5fe6-4ae8-a425-23208a33537b-kube-api-access-wmslq\") pod \"maas-api-revoke-keys-e2e-api-b-ca3f31-p656b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:21.983832 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:21.983794    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:22.361399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:22.361372    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"]
Nov 03 18:19:22.363005 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:19:22.362977    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c141b70_5fe6_4ae8_a425_23208a33537b.slice/crio-805a6be4de5004a19f5da057652f1e48bf80f38a7c140e498c74ab9496fba304 WatchSource:0}: Error finding container 805a6be4de5004a19f5da057652f1e48bf80f38a7c140e498c74ab9496fba304: Status 404 returned error can't find the container with id 805a6be4de5004a19f5da057652f1e48bf80f38a7c140e498c74ab9496fba304
Nov 03 18:19:22.892927 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:22.892826    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b" event={"ID":"1c141b70-5fe6-4ae8-a425-23208a33537b","Type":"ContainerStarted","Data":"da1490e3f10ab95d41e03691cc531612c77ece059b6b168446151f3a0c54ee40"}
Nov 03 18:19:22.892927 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:22.892901    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b" event={"ID":"1c141b70-5fe6-4ae8-a425-23208a33537b","Type":"ContainerStarted","Data":"805a6be4de5004a19f5da057652f1e48bf80f38a7c140e498c74ab9496fba304"}
Nov 03 18:19:23.013432 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:23.013378    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b" podStartSLOduration=2.013365962 podStartE2EDuration="2.013365962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:19:23.013166351 +0000 UTC m=+2742.472337338" watchObservedRunningTime="2026-11-03 18:19:23.013365962 +0000 UTC m=+2742.472536945"
Nov 03 18:19:23.901010 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:23.900974    4493 generic.go:356] "Generic (PLEG): container finished" podID="1c141b70-5fe6-4ae8-a425-23208a33537b" containerID="da1490e3f10ab95d41e03691cc531612c77ece059b6b168446151f3a0c54ee40" exitCode=0
Nov 03 18:19:23.901430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:23.901065    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b" event={"ID":"1c141b70-5fe6-4ae8-a425-23208a33537b","Type":"ContainerDied","Data":"da1490e3f10ab95d41e03691cc531612c77ece059b6b168446151f3a0c54ee40"}
Nov 03 18:19:25.065455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.065422    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:25.237451 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.237413    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-openshift-ca\") pod \"1c141b70-5fe6-4ae8-a425-23208a33537b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") "
Nov 03 18:19:25.237623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.237522    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-xks-ca\") pod \"1c141b70-5fe6-4ae8-a425-23208a33537b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") "
Nov 03 18:19:25.237623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.237556    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c141b70-5fe6-4ae8-a425-23208a33537b-kube-api-access-wmslq\" (UniqueName: \"kubernetes.io/projected/1c141b70-5fe6-4ae8-a425-23208a33537b-kube-api-access-wmslq\") pod \"1c141b70-5fe6-4ae8-a425-23208a33537b\" (UID: \"1c141b70-5fe6-4ae8-a425-23208a33537b\") "
Nov 03 18:19:25.237851 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.237824    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-openshift-ca" pod "1c141b70-5fe6-4ae8-a425-23208a33537b" (UID: "1c141b70-5fe6-4ae8-a425-23208a33537b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:25.239715 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.239688    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-xks-ca" pod "1c141b70-5fe6-4ae8-a425-23208a33537b" (UID: "1c141b70-5fe6-4ae8-a425-23208a33537b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:19:25.239861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.239841    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c141b70-5fe6-4ae8-a425-23208a33537b-kube-api-access-wmslq" pod "1c141b70-5fe6-4ae8-a425-23208a33537b" (UID: "1c141b70-5fe6-4ae8-a425-23208a33537b"). InnerVolumeSpecName "kube-api-access-wmslq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:25.338834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.338801    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:25.338834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.338828    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmslq\" (UniqueName: \"kubernetes.io/projected/1c141b70-5fe6-4ae8-a425-23208a33537b-kube-api-access-wmslq\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:25.338834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.338839    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c141b70-5fe6-4ae8-a425-23208a33537b-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:25.916031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.916005    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"
Nov 03 18:19:25.916031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.916015    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b" event={"ID":"1c141b70-5fe6-4ae8-a425-23208a33537b","Type":"ContainerDied","Data":"805a6be4de5004a19f5da057652f1e48bf80f38a7c140e498c74ab9496fba304"}
Nov 03 18:19:25.916264 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:25.916070    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805a6be4de5004a19f5da057652f1e48bf80f38a7c140e498c74ab9496fba304"
Nov 03 18:19:33.189847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:33.189804    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"]
Nov 03 18:19:33.301127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:33.301088    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"]
Nov 03 18:19:36.993113 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:19:36.993075    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:19:38.190501 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.190465    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9" podUID="d37a77a9-db23-40a1-9472-0494d16783e4" containerName="payload-processing" containerID="cri-o://3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea" gracePeriod=25
Nov 03 18:19:38.301789 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.301727    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8" podUID="6ae3ce93-e6ea-4b75-a364-96e7c6458a63" containerName="payload-pre-processing" containerID="cri-o://558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af" gracePeriod=25
Nov 03 18:19:38.449261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.449175    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:19:38.490585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.490550    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37a77a9-db23-40a1-9472-0494d16783e4-kube-api-access-w72pf\" (UniqueName: \"kubernetes.io/projected/d37a77a9-db23-40a1-9472-0494d16783e4-kube-api-access-w72pf\") pod \"d37a77a9-db23-40a1-9472-0494d16783e4\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") "
Nov 03 18:19:38.490806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.490788    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37a77a9-db23-40a1-9472-0494d16783e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a77a9-db23-40a1-9472-0494d16783e4-plugins-config-volume\") pod \"d37a77a9-db23-40a1-9472-0494d16783e4\" (UID: \"d37a77a9-db23-40a1-9472-0494d16783e4\") "
Nov 03 18:19:38.491171 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.491146    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37a77a9-db23-40a1-9472-0494d16783e4-plugins-config-volume" pod "d37a77a9-db23-40a1-9472-0494d16783e4" (UID: "d37a77a9-db23-40a1-9472-0494d16783e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:38.491324 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.491292    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a77a9-db23-40a1-9472-0494d16783e4-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.492970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.492937    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37a77a9-db23-40a1-9472-0494d16783e4-kube-api-access-w72pf" pod "d37a77a9-db23-40a1-9472-0494d16783e4" (UID: "d37a77a9-db23-40a1-9472-0494d16783e4"). InnerVolumeSpecName "kube-api-access-w72pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:38.552368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.552342    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:19:38.592637 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.592599    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-plugins-config-volume\") pod \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") "
Nov 03 18:19:38.592819 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.592714    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-kube-api-access-wblpz\" (UniqueName: \"kubernetes.io/projected/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-kube-api-access-wblpz\") pod \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\" (UID: \"6ae3ce93-e6ea-4b75-a364-96e7c6458a63\") "
Nov 03 18:19:38.593084 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.593060    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-plugins-config-volume" pod "6ae3ce93-e6ea-4b75-a364-96e7c6458a63" (UID: "6ae3ce93-e6ea-4b75-a364-96e7c6458a63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:19:38.593220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.593203    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.593290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.593223    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72pf\" (UniqueName: \"kubernetes.io/projected/d37a77a9-db23-40a1-9472-0494d16783e4-kube-api-access-w72pf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.595394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.595370    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-kube-api-access-wblpz" pod "6ae3ce93-e6ea-4b75-a364-96e7c6458a63" (UID: "6ae3ce93-e6ea-4b75-a364-96e7c6458a63"). InnerVolumeSpecName "kube-api-access-wblpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:19:38.694369 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.694329    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wblpz\" (UniqueName: \"kubernetes.io/projected/6ae3ce93-e6ea-4b75-a364-96e7c6458a63-kube-api-access-wblpz\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:19:38.993270 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.993172    4493 generic.go:356] "Generic (PLEG): container finished" podID="6ae3ce93-e6ea-4b75-a364-96e7c6458a63" containerID="558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af" exitCode=0
Nov 03 18:19:38.993270 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.993216    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"
Nov 03 18:19:38.993513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.993261    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8" event={"ID":"6ae3ce93-e6ea-4b75-a364-96e7c6458a63","Type":"ContainerDied","Data":"558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af"}
Nov 03 18:19:38.993513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.993316    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8" event={"ID":"6ae3ce93-e6ea-4b75-a364-96e7c6458a63","Type":"ContainerDied","Data":"9856f86703fee62e8f21aab21007a8690332e68725f1fe77962d28d7291b8345"}
Nov 03 18:19:38.993513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.993340    4493 scope.go:122] "RemoveContainer" containerID="558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af"
Nov 03 18:19:38.995064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.995005    4493 generic.go:356] "Generic (PLEG): container finished" podID="d37a77a9-db23-40a1-9472-0494d16783e4" containerID="3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea" exitCode=0
Nov 03 18:19:38.995186 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.995076    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"
Nov 03 18:19:38.995186 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.995089    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9" event={"ID":"d37a77a9-db23-40a1-9472-0494d16783e4","Type":"ContainerDied","Data":"3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea"}
Nov 03 18:19:38.995186 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:38.995134    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9" event={"ID":"d37a77a9-db23-40a1-9472-0494d16783e4","Type":"ContainerDied","Data":"1b1462ddaa7fd68c85af6a06afb4ef8d5733a3a7f1d82f06d9d5275ca62c52a5"}
Nov 03 18:19:39.005986 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.005964    4493 scope.go:122] "RemoveContainer" containerID="558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af"
Nov 03 18:19:39.006338 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:19:39.006315    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af\": container with ID starting with 558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af not found: ID does not exist" containerID="558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af"
Nov 03 18:19:39.006412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.006349    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af"} err="failed to get container status \"558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af\": rpc error: code = NotFound desc = could not find container \"558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af\": container with ID starting with 558b76bb7f192599e5b5d978cc885217ac65cc896b9dbe2a7053473d4fc2d6af not found: ID does not exist"
Nov 03 18:19:39.006412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.006367    4493 scope.go:122] "RemoveContainer" containerID="3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea"
Nov 03 18:19:39.015443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.015424    4493 scope.go:122] "RemoveContainer" containerID="3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea"
Nov 03 18:19:39.015750 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:19:39.015728    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea\": container with ID starting with 3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea not found: ID does not exist" containerID="3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea"
Nov 03 18:19:39.015863 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.015757    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea"} err="failed to get container status \"3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea\": rpc error: code = NotFound desc = could not find container \"3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea\": container with ID starting with 3064fe39f2803a5ab17fb2c4847e8a94633bdbeed0aefa4a7e3392279bff9bea not found: ID does not exist"
Nov 03 18:19:39.163782 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.163738    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"]
Nov 03 18:19:39.166428 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.166402    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ca3f31-6b69fd66cf-xbgq9"]
Nov 03 18:19:39.280087 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.280003    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"]
Nov 03 18:19:39.286118 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:39.286092    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ca3f31-5cc4dfbf4c-st6b8"]
Nov 03 18:19:40.712365 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:40.712333    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae3ce93-e6ea-4b75-a364-96e7c6458a63" path="/var/lib/kubelet/pods/6ae3ce93-e6ea-4b75-a364-96e7c6458a63/volumes"
Nov 03 18:19:40.712720 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:40.712686    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37a77a9-db23-40a1-9472-0494d16783e4" path="/var/lib/kubelet/pods/d37a77a9-db23-40a1-9472-0494d16783e4/volumes"
Nov 03 18:19:43.047844 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:43.047811    4493 scope.go:122] "RemoveContainer" containerID="20e19a4868045dda385fa6315f681d9335bfa36b9819c817c943ffb6dc732cb0"
Nov 03 18:19:43.057040 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:43.057019    4493 scope.go:122] "RemoveContainer" containerID="f8330a81e8dbab21766de6a3a924149fe84f5997c84ab7a7a7fccc902f433ba2"
Nov 03 18:19:43.065482 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:43.065461    4493 scope.go:122] "RemoveContainer" containerID="2105fbe9344fb2acc8e35af5802a2ebeeed7bb52394cd69e2349fe075c28581a"
Nov 03 18:19:43.074125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:43.074104    4493 scope.go:122] "RemoveContainer" containerID="1d8e04b3e4460ae5b70bf59d74636b9cf8a343a8a53296e3e55a55516f3c48cf"
Nov 03 18:19:54.437424 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:54.437385    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"]
Nov 03 18:19:54.488040 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:54.488005    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-44769b-grfrp"]
Nov 03 18:19:54.719761 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:19:54.719722    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d8ef9a-b646-4705-adfa-b4ea586eb681" path="/var/lib/kubelet/pods/73d8ef9a-b646-4705-adfa-b4ea586eb681/volumes"
Nov 03 18:20:03.626262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626224    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"]
Nov 03 18:20:03.626653 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626640    4493 cpu_manager.go:414] "containerMap: removing container" podUID="d37a77a9-db23-40a1-9472-0494d16783e4" containerName="payload-processing"
Nov 03 18:20:03.626653 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626651    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37a77a9-db23-40a1-9472-0494d16783e4" containerName="payload-processing"
Nov 03 18:20:03.626724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626663    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ae3ce93-e6ea-4b75-a364-96e7c6458a63" containerName="payload-pre-processing"
Nov 03 18:20:03.626724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626668    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae3ce93-e6ea-4b75-a364-96e7c6458a63" containerName="payload-pre-processing"
Nov 03 18:20:03.626724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626679    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1c141b70-5fe6-4ae8-a425-23208a33537b" containerName="revoke-keys"
Nov 03 18:20:03.626724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626683    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c141b70-5fe6-4ae8-a425-23208a33537b" containerName="revoke-keys"
Nov 03 18:20:03.626864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626758    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37a77a9-db23-40a1-9472-0494d16783e4" containerName="payload-processing"
Nov 03 18:20:03.626864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626766    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae3ce93-e6ea-4b75-a364-96e7c6458a63" containerName="payload-pre-processing"
Nov 03 18:20:03.626864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.626774    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c141b70-5fe6-4ae8-a425-23208a33537b" containerName="revoke-keys"
Nov 03 18:20:03.632305 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.632285    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.680948 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.680909    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndvpz\" (UniqueName: \"kubernetes.io/projected/fa034827-8abd-4269-b345-e4075b230588-kube-api-access-ndvpz\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.681133 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.680969    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa034827-8abd-4269-b345-e4075b230588-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.681133 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.681018    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa034827-8abd-4269-b345-e4075b230588-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.690015 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.689982    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"]
Nov 03 18:20:03.736970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.736942    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:20:03.736970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.736954    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:20:03.737164 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.736952    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:20:03.737164 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.736996    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:20:03.782242 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.782209    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndvpz\" (UniqueName: \"kubernetes.io/projected/fa034827-8abd-4269-b345-e4075b230588-kube-api-access-ndvpz\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.782408 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.782267    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa034827-8abd-4269-b345-e4075b230588-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.782408 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.782283    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa034827-8abd-4269-b345-e4075b230588-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.782948 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.782925    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa034827-8abd-4269-b345-e4075b230588-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.784675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.784655    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa034827-8abd-4269-b345-e4075b230588-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.847685 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.847649    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndvpz\" (UniqueName: \"kubernetes.io/projected/fa034827-8abd-4269-b345-e4075b230588-kube-api-access-ndvpz\") pod \"maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:03.945851 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:03.945783    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:04.327024 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:04.326999    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"]
Nov 03 18:20:04.329013 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:20:04.328984    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa034827_8abd_4269_b345_e4075b230588.slice/crio-d9a432619eb5a8b2754e11901c17d36cf98bee9da8ba314b32873da00dc23cd8 WatchSource:0}: Error finding container d9a432619eb5a8b2754e11901c17d36cf98bee9da8ba314b32873da00dc23cd8: Status 404 returned error can't find the container with id d9a432619eb5a8b2754e11901c17d36cf98bee9da8ba314b32873da00dc23cd8
Nov 03 18:20:05.151953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:05.151919    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh" event={"ID":"fa034827-8abd-4269-b345-e4075b230588","Type":"ContainerStarted","Data":"8cb47b28168b58118df98e42f98fd32d73de0a5ab28c7665590cc57bbad7e424"}
Nov 03 18:20:05.152353 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:05.151996    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh" event={"ID":"fa034827-8abd-4269-b345-e4075b230588","Type":"ContainerStarted","Data":"d9a432619eb5a8b2754e11901c17d36cf98bee9da8ba314b32873da00dc23cd8"}
Nov 03 18:20:05.276420 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:05.276371    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh" podStartSLOduration=2.276358063 podStartE2EDuration="2.276358063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:20:05.274370767 +0000 UTC m=+2784.733541807" watchObservedRunningTime="2026-11-03 18:20:05.276358063 +0000 UTC m=+2784.735529050"
Nov 03 18:20:06.158459 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:06.158426    4493 generic.go:356] "Generic (PLEG): container finished" podID="fa034827-8abd-4269-b345-e4075b230588" containerID="8cb47b28168b58118df98e42f98fd32d73de0a5ab28c7665590cc57bbad7e424" exitCode=0
Nov 03 18:20:06.158847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:06.158517    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh" event={"ID":"fa034827-8abd-4269-b345-e4075b230588","Type":"ContainerDied","Data":"8cb47b28168b58118df98e42f98fd32d73de0a5ab28c7665590cc57bbad7e424"}
Nov 03 18:20:06.991576 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:20:06.991539    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:20:07.298200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.298174    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:07.313954 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.313928    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa034827-8abd-4269-b345-e4075b230588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa034827-8abd-4269-b345-e4075b230588-maas-api-xks-ca\") pod \"fa034827-8abd-4269-b345-e4075b230588\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") "
Nov 03 18:20:07.314087 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.313993    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa034827-8abd-4269-b345-e4075b230588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa034827-8abd-4269-b345-e4075b230588-maas-api-openshift-ca\") pod \"fa034827-8abd-4269-b345-e4075b230588\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") "
Nov 03 18:20:07.314133 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.314084    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa034827-8abd-4269-b345-e4075b230588-kube-api-access-ndvpz\" (UniqueName: \"kubernetes.io/projected/fa034827-8abd-4269-b345-e4075b230588-kube-api-access-ndvpz\") pod \"fa034827-8abd-4269-b345-e4075b230588\" (UID: \"fa034827-8abd-4269-b345-e4075b230588\") "
Nov 03 18:20:07.314353 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.314327    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa034827-8abd-4269-b345-e4075b230588-maas-api-openshift-ca" pod "fa034827-8abd-4269-b345-e4075b230588" (UID: "fa034827-8abd-4269-b345-e4075b230588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:07.316127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.316106    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa034827-8abd-4269-b345-e4075b230588-maas-api-xks-ca" pod "fa034827-8abd-4269-b345-e4075b230588" (UID: "fa034827-8abd-4269-b345-e4075b230588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:07.316227 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.316209    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa034827-8abd-4269-b345-e4075b230588-kube-api-access-ndvpz" pod "fa034827-8abd-4269-b345-e4075b230588" (UID: "fa034827-8abd-4269-b345-e4075b230588"). InnerVolumeSpecName "kube-api-access-ndvpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:07.414959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.414930    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndvpz\" (UniqueName: \"kubernetes.io/projected/fa034827-8abd-4269-b345-e4075b230588-kube-api-access-ndvpz\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:07.414959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.414956    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa034827-8abd-4269-b345-e4075b230588-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:07.414959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:07.414965    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa034827-8abd-4269-b345-e4075b230588-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:08.036829 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:08.036792    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"]
Nov 03 18:20:08.048778 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:08.048749    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"]
Nov 03 18:20:08.173654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:08.173621    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"
Nov 03 18:20:08.173831 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:08.173618    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh" event={"ID":"fa034827-8abd-4269-b345-e4075b230588","Type":"ContainerDied","Data":"d9a432619eb5a8b2754e11901c17d36cf98bee9da8ba314b32873da00dc23cd8"}
Nov 03 18:20:08.173831 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:08.173760    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9a432619eb5a8b2754e11901c17d36cf98bee9da8ba314b32873da00dc23cd8"
Nov 03 18:20:15.830543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.830510    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"]
Nov 03 18:20:15.831029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.830951    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fa034827-8abd-4269-b345-e4075b230588" containerName="revoke-keys"
Nov 03 18:20:15.831029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.830964    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa034827-8abd-4269-b345-e4075b230588" containerName="revoke-keys"
Nov 03 18:20:15.831029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.831026    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa034827-8abd-4269-b345-e4075b230588" containerName="revoke-keys"
Nov 03 18:20:15.834135 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.834120    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:15.893660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.893629    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"]
Nov 03 18:20:15.902959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.902928    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:15.903116 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.903002    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:15.903171 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.903152    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6gt\" (UniqueName: \"kubernetes.io/projected/deee988e-fdea-46c6-9b4f-ff9582203d4b-kube-api-access-xs6gt\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:15.941287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.941262    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:20:15.941438 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.941287    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:20:15.941438 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.941287    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:20:15.941663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:15.941646    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:20:16.004448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.004420    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:16.004587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.004462    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:16.004587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.004504    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6gt\" (UniqueName: \"kubernetes.io/projected/deee988e-fdea-46c6-9b4f-ff9582203d4b-kube-api-access-xs6gt\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:16.005144 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.005126    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:16.007005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.006985    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:16.068296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.068270    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6gt\" (UniqueName: \"kubernetes.io/projected/deee988e-fdea-46c6-9b4f-ff9582203d4b-kube-api-access-xs6gt\") pod \"maas-api-revoke-sub-05af8a05-snq8t\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:16.147078 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.147005    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:16.525041 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:16.525008    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"]
Nov 03 18:20:16.525441 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:20:16.525416    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeee988e_fdea_46c6_9b4f_ff9582203d4b.slice/crio-22a9047e99d76405a09fd6b64712110052859521a8d471bdf84038f1126a3d85 WatchSource:0}: Error finding container 22a9047e99d76405a09fd6b64712110052859521a8d471bdf84038f1126a3d85: Status 404 returned error can't find the container with id 22a9047e99d76405a09fd6b64712110052859521a8d471bdf84038f1126a3d85
Nov 03 18:20:17.229410 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:17.229366    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t" event={"ID":"deee988e-fdea-46c6-9b4f-ff9582203d4b","Type":"ContainerStarted","Data":"6738bf90cedc39723bbc1cd57b8fdf316c72f2599a6f7002e90cc3e97c4f6697"}
Nov 03 18:20:17.229858 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:17.229432    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t" event={"ID":"deee988e-fdea-46c6-9b4f-ff9582203d4b","Type":"ContainerStarted","Data":"22a9047e99d76405a09fd6b64712110052859521a8d471bdf84038f1126a3d85"}
Nov 03 18:20:17.373466 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:17.373402    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t" podStartSLOduration=2.373387237 podStartE2EDuration="2.373387237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:20:17.350040668 +0000 UTC m=+2796.809211655" watchObservedRunningTime="2026-11-03 18:20:17.373387237 +0000 UTC m=+2796.832558222"
Nov 03 18:20:18.236635 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:18.236596    4493 generic.go:356] "Generic (PLEG): container finished" podID="deee988e-fdea-46c6-9b4f-ff9582203d4b" containerID="6738bf90cedc39723bbc1cd57b8fdf316c72f2599a6f7002e90cc3e97c4f6697" exitCode=0
Nov 03 18:20:18.237086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:18.236680    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t" event={"ID":"deee988e-fdea-46c6-9b4f-ff9582203d4b","Type":"ContainerDied","Data":"6738bf90cedc39723bbc1cd57b8fdf316c72f2599a6f7002e90cc3e97c4f6697"}
Nov 03 18:20:19.393129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.393104    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:19.453477 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.453441    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deee988e-fdea-46c6-9b4f-ff9582203d4b-kube-api-access-xs6gt\" (UniqueName: \"kubernetes.io/projected/deee988e-fdea-46c6-9b4f-ff9582203d4b-kube-api-access-xs6gt\") pod \"deee988e-fdea-46c6-9b4f-ff9582203d4b\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") "
Nov 03 18:20:19.453668 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.453502    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-xks-ca\") pod \"deee988e-fdea-46c6-9b4f-ff9582203d4b\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") "
Nov 03 18:20:19.453668 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.453566    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-openshift-ca\") pod \"deee988e-fdea-46c6-9b4f-ff9582203d4b\" (UID: \"deee988e-fdea-46c6-9b4f-ff9582203d4b\") "
Nov 03 18:20:19.454008 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.453980    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-openshift-ca" pod "deee988e-fdea-46c6-9b4f-ff9582203d4b" (UID: "deee988e-fdea-46c6-9b4f-ff9582203d4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:19.455817 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.455789    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-xks-ca" pod "deee988e-fdea-46c6-9b4f-ff9582203d4b" (UID: "deee988e-fdea-46c6-9b4f-ff9582203d4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:19.455939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.455867    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deee988e-fdea-46c6-9b4f-ff9582203d4b-kube-api-access-xs6gt" pod "deee988e-fdea-46c6-9b4f-ff9582203d4b" (UID: "deee988e-fdea-46c6-9b4f-ff9582203d4b"). InnerVolumeSpecName "kube-api-access-xs6gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:19.554850 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.554770    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6gt\" (UniqueName: \"kubernetes.io/projected/deee988e-fdea-46c6-9b4f-ff9582203d4b-kube-api-access-xs6gt\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:19.554850 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.554802    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:19.554850 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:19.554811    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deee988e-fdea-46c6-9b4f-ff9582203d4b-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:20.249038 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:20.249003    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t" event={"ID":"deee988e-fdea-46c6-9b4f-ff9582203d4b","Type":"ContainerDied","Data":"22a9047e99d76405a09fd6b64712110052859521a8d471bdf84038f1126a3d85"}
Nov 03 18:20:20.249222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:20.249043    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"
Nov 03 18:20:20.249222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:20.249053    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22a9047e99d76405a09fd6b64712110052859521a8d471bdf84038f1126a3d85"
Nov 03 18:20:23.057815 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.057776    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main" containerID="cri-o://2daf9b8eb31debbfc102d0cf9be65d5329abd740835630c45a5daa8697ab62c8" gracePeriod=45
Nov 03 18:20:23.070341 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.070299    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" containerID="cri-o://523d2fdbe8d685914c1b65f5564caa6650c8da6c678fab9ca57d8a254383e3f5" gracePeriod=45
Nov 03 18:20:23.108173 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.108137    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": dial tcp 10.133.0.38:8000: connect: connection refused"
Nov 03 18:20:23.273662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.273624    4493 generic.go:356] "Generic (PLEG): container finished" podID="c4780426-c057-47be-af63-4083a7fc8973" containerID="2daf9b8eb31debbfc102d0cf9be65d5329abd740835630c45a5daa8697ab62c8" exitCode=0
Nov 03 18:20:23.273839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.273709    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" event={"ID":"c4780426-c057-47be-af63-4083a7fc8973","Type":"ContainerDied","Data":"2daf9b8eb31debbfc102d0cf9be65d5329abd740835630c45a5daa8697ab62c8"}
Nov 03 18:20:23.275755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.275732    4493 generic.go:356] "Generic (PLEG): container finished" podID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerID="523d2fdbe8d685914c1b65f5564caa6650c8da6c678fab9ca57d8a254383e3f5" exitCode=0
Nov 03 18:20:23.275917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.275764    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" event={"ID":"38c08f57-c68d-421f-980d-f66efb8c53f0","Type":"ContainerDied","Data":"523d2fdbe8d685914c1b65f5564caa6650c8da6c678fab9ca57d8a254383e3f5"}
Nov 03 18:20:23.335361 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.335336    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 18:20:23.356286 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.356260    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 18:20:23.494277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494238    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-home\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-home\") pod \"c4780426-c057-47be-af63-4083a7fc8973\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") "
Nov 03 18:20:23.494473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494303    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4780426-c057-47be-af63-4083a7fc8973-kube-api-access-mggpt\" (UniqueName: \"kubernetes.io/projected/c4780426-c057-47be-af63-4083a7fc8973-kube-api-access-mggpt\") pod \"c4780426-c057-47be-af63-4083a7fc8973\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") "
Nov 03 18:20:23.494473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494346    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-model-cache\") pod \"38c08f57-c68d-421f-980d-f66efb8c53f0\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") "
Nov 03 18:20:23.494473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494399    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-tmp-dir\") pod \"38c08f57-c68d-421f-980d-f66efb8c53f0\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") "
Nov 03 18:20:23.494473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494442    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-model-cache\") pod \"c4780426-c057-47be-af63-4083a7fc8973\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") "
Nov 03 18:20:23.494711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494513    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-home\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-home\") pod \"38c08f57-c68d-421f-980d-f66efb8c53f0\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") "
Nov 03 18:20:23.494711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494543    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-dshm\") pod \"38c08f57-c68d-421f-980d-f66efb8c53f0\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") "
Nov 03 18:20:23.494711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494547    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-home" pod "c4780426-c057-47be-af63-4083a7fc8973" (UID: "c4780426-c057-47be-af63-4083a7fc8973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.494711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494590    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-dshm\") pod \"c4780426-c057-47be-af63-4083a7fc8973\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") "
Nov 03 18:20:23.494711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494642    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c08f57-c68d-421f-980d-f66efb8c53f0-kube-api-access-srjtj\" (UniqueName: \"kubernetes.io/projected/38c08f57-c68d-421f-980d-f66efb8c53f0-kube-api-access-srjtj\") pod \"38c08f57-c68d-421f-980d-f66efb8c53f0\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") "
Nov 03 18:20:23.494711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494682    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c08f57-c68d-421f-980d-f66efb8c53f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/38c08f57-c68d-421f-980d-f66efb8c53f0-tls-certs\") pod \"38c08f57-c68d-421f-980d-f66efb8c53f0\" (UID: \"38c08f57-c68d-421f-980d-f66efb8c53f0\") "
Nov 03 18:20:23.494711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494711    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4780426-c057-47be-af63-4083a7fc8973-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4780426-c057-47be-af63-4083a7fc8973-tls-certs\") pod \"c4780426-c057-47be-af63-4083a7fc8973\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") "
Nov 03 18:20:23.495152 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494705    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-model-cache" pod "38c08f57-c68d-421f-980d-f66efb8c53f0" (UID: "38c08f57-c68d-421f-980d-f66efb8c53f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.495152 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494717    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-tmp-dir" pod "38c08f57-c68d-421f-980d-f66efb8c53f0" (UID: "38c08f57-c68d-421f-980d-f66efb8c53f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.495152 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494778    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-home" pod "38c08f57-c68d-421f-980d-f66efb8c53f0" (UID: "38c08f57-c68d-421f-980d-f66efb8c53f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.495152 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.494856    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-tmp-dir\") pod \"c4780426-c057-47be-af63-4083a7fc8973\" (UID: \"c4780426-c057-47be-af63-4083a7fc8973\") "
Nov 03 18:20:23.495375 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495185    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-model-cache" pod "c4780426-c057-47be-af63-4083a7fc8973" (UID: "c4780426-c057-47be-af63-4083a7fc8973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.495506 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495463    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-tmp-dir" pod "c4780426-c057-47be-af63-4083a7fc8973" (UID: "c4780426-c057-47be-af63-4083a7fc8973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.495704 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495681    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.495805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495710    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.495805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495726    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.495805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495740    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.495805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495752    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.495805 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.495767    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.497806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.497771    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c08f57-c68d-421f-980d-f66efb8c53f0-kube-api-access-srjtj" pod "38c08f57-c68d-421f-980d-f66efb8c53f0" (UID: "38c08f57-c68d-421f-980d-f66efb8c53f0"). InnerVolumeSpecName "kube-api-access-srjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:23.497947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.497928    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-dshm" pod "c4780426-c057-47be-af63-4083a7fc8973" (UID: "c4780426-c057-47be-af63-4083a7fc8973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.498042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.497943    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-dshm" pod "38c08f57-c68d-421f-980d-f66efb8c53f0" (UID: "38c08f57-c68d-421f-980d-f66efb8c53f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:23.498042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.498025    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c08f57-c68d-421f-980d-f66efb8c53f0-tls-certs" pod "38c08f57-c68d-421f-980d-f66efb8c53f0" (UID: "38c08f57-c68d-421f-980d-f66efb8c53f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:23.498576 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.498503    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4780426-c057-47be-af63-4083a7fc8973-tls-certs" pod "c4780426-c057-47be-af63-4083a7fc8973" (UID: "c4780426-c057-47be-af63-4083a7fc8973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:23.499067 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.499042    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4780426-c057-47be-af63-4083a7fc8973-kube-api-access-mggpt" pod "c4780426-c057-47be-af63-4083a7fc8973" (UID: "c4780426-c057-47be-af63-4083a7fc8973"). InnerVolumeSpecName "kube-api-access-mggpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:23.597641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.597565    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38c08f57-c68d-421f-980d-f66efb8c53f0-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.597641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.597603    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4780426-c057-47be-af63-4083a7fc8973-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.597641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.597621    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srjtj\" (UniqueName: \"kubernetes.io/projected/38c08f57-c68d-421f-980d-f66efb8c53f0-kube-api-access-srjtj\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.597641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.597638    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38c08f57-c68d-421f-980d-f66efb8c53f0-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.597943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.597654    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4780426-c057-47be-af63-4083a7fc8973-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:23.597943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:23.597667    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mggpt\" (UniqueName: \"kubernetes.io/projected/c4780426-c057-47be-af63-4083a7fc8973-kube-api-access-mggpt\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:24.283301 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.283271    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"
Nov 03 18:20:24.283778 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.283297    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9" event={"ID":"38c08f57-c68d-421f-980d-f66efb8c53f0","Type":"ContainerDied","Data":"f98fe38c1a2c8d1572e9f3bdee8d7391bab53c0f5ddfc1e899a01fbd81df5014"}
Nov 03 18:20:24.283778 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.283348    4493 scope.go:122] "RemoveContainer" containerID="523d2fdbe8d685914c1b65f5564caa6650c8da6c678fab9ca57d8a254383e3f5"
Nov 03 18:20:24.284940 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.284919    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk" event={"ID":"c4780426-c057-47be-af63-4083a7fc8973","Type":"ContainerDied","Data":"94f6dd846bd7c435876535adb04ea987761f50a20d9e4681f9742fa3a7c5d52b"}
Nov 03 18:20:24.284940 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.284935    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"
Nov 03 18:20:24.294794 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.294769    4493 scope.go:122] "RemoveContainer" containerID="2daf9b8eb31debbfc102d0cf9be65d5329abd740835630c45a5daa8697ab62c8"
Nov 03 18:20:24.453768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.453679    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"]
Nov 03 18:20:24.461704 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.461671    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/simulator-w6-f7d463-kserve-f497bc67c-q4jd9"]
Nov 03 18:20:24.575939 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.575901    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"]
Nov 03 18:20:24.580847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.580816    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7d463/premium-w6-f7d463-kserve-6f5c9784dd-n9zwk"]
Nov 03 18:20:24.713712 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.713683    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" path="/var/lib/kubelet/pods/38c08f57-c68d-421f-980d-f66efb8c53f0/volumes"
Nov 03 18:20:24.714339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:24.714315    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4780426-c057-47be-af63-4083a7fc8973" path="/var/lib/kubelet/pods/c4780426-c057-47be-af63-4083a7fc8973/volumes"
Nov 03 18:20:25.583803 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.583769    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"]
Nov 03 18:20:25.584214 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584201    4493 cpu_manager.go:414] "containerMap: removing container" podUID="deee988e-fdea-46c6-9b4f-ff9582203d4b" containerName="revoke-keys"
Nov 03 18:20:25.584214 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584212    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deee988e-fdea-46c6-9b4f-ff9582203d4b" containerName="revoke-keys"
Nov 03 18:20:25.584289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584235    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main"
Nov 03 18:20:25.584289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584240    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main"
Nov 03 18:20:25.584289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584248    4493 cpu_manager.go:414] "containerMap: removing container" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main"
Nov 03 18:20:25.584289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584253    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main"
Nov 03 18:20:25.584416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584318    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="deee988e-fdea-46c6-9b4f-ff9582203d4b" containerName="revoke-keys"
Nov 03 18:20:25.584416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584330    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4780426-c057-47be-af63-4083a7fc8973" containerName="main"
Nov 03 18:20:25.584416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.584337    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c08f57-c68d-421f-980d-f66efb8c53f0" containerName="main"
Nov 03 18:20:25.588654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.588635    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.644654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.644618    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"]
Nov 03 18:20:25.648989 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.648962    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.671934 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.671908    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"]
Nov 03 18:20:25.692390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.692360    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:20:25.692555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.692439    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:20:25.692555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.692467    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:20:25.692555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.692520    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:20:25.714572 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.714545    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"]
Nov 03 18:20:25.722255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.722223    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.722413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.722361    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqd7\" (UniqueName: \"kubernetes.io/projected/0448e549-eef3-4706-81ac-f26c44eca459-kube-api-access-btqd7\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.722486 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.722463    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjk8q\" (UniqueName: \"kubernetes.io/projected/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-kube-api-access-wjk8q\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.722547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.722526    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.722649 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.722634    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.722716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.722668    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.824124 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.824090    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqd7\" (UniqueName: \"kubernetes.io/projected/0448e549-eef3-4706-81ac-f26c44eca459-kube-api-access-btqd7\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.824344 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.824160    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjk8q\" (UniqueName: \"kubernetes.io/projected/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-kube-api-access-wjk8q\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.824344 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.824210    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.824344 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.824297    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.824344 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.824322    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.824580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.824371    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.824955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.824927    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.825104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.825013    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.833952 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.827837    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.833952 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.827829    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.887826 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.887791    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjk8q\" (UniqueName: \"kubernetes.io/projected/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-kube-api-access-wjk8q\") pod \"maas-api-revoke-sub-ac7bb928-hdv4d\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:25.887995 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.887792    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqd7\" (UniqueName: \"kubernetes.io/projected/0448e549-eef3-4706-81ac-f26c44eca459-kube-api-access-btqd7\") pod \"maas-api-revoke-sub-0fa601de-8w2zs\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.901377 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.901354    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:25.968350 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:25.968093    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:26.283215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:26.283187    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"]
Nov 03 18:20:26.284375 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:20:26.284350    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0448e549_eef3_4706_81ac_f26c44eca459.slice/crio-410fda888d455a3273faca676aa8af519f3cd0ae6e1268ad6d1a18ed2aa6e1a4 WatchSource:0}: Error finding container 410fda888d455a3273faca676aa8af519f3cd0ae6e1268ad6d1a18ed2aa6e1a4: Status 404 returned error can't find the container with id 410fda888d455a3273faca676aa8af519f3cd0ae6e1268ad6d1a18ed2aa6e1a4
Nov 03 18:20:26.299762 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:26.299732    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs" event={"ID":"0448e549-eef3-4706-81ac-f26c44eca459","Type":"ContainerStarted","Data":"410fda888d455a3273faca676aa8af519f3cd0ae6e1268ad6d1a18ed2aa6e1a4"}
Nov 03 18:20:26.397490 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:26.397463    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"]
Nov 03 18:20:26.398448 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:20:26.398416    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebc30cd_e785_4a2a_88ed_1f6143bc0bdd.slice/crio-14809fc519d2913f70b175de42564e5fa0748fffe1982334f6588da1856db16f WatchSource:0}: Error finding container 14809fc519d2913f70b175de42564e5fa0748fffe1982334f6588da1856db16f: Status 404 returned error can't find the container with id 14809fc519d2913f70b175de42564e5fa0748fffe1982334f6588da1856db16f
Nov 03 18:20:26.499432 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:26.499404    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"]
Nov 03 18:20:26.503997 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:26.503971    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ca3f31-p656b"]
Nov 03 18:20:26.712789 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:26.712747    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c141b70-5fe6-4ae8-a425-23208a33537b" path="/var/lib/kubelet/pods/1c141b70-5fe6-4ae8-a425-23208a33537b/volumes"
Nov 03 18:20:27.311919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:27.311886    4493 generic.go:356] "Generic (PLEG): container finished" podID="0448e549-eef3-4706-81ac-f26c44eca459" containerID="a4223d8a40916a8f9e55fec9e979d893057ab8ae58abab45a378aec9657a6678" exitCode=0
Nov 03 18:20:27.311919 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:27.311906    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs" event={"ID":"0448e549-eef3-4706-81ac-f26c44eca459","Type":"ContainerDied","Data":"a4223d8a40916a8f9e55fec9e979d893057ab8ae58abab45a378aec9657a6678"}
Nov 03 18:20:27.313546 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:27.313520    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d" event={"ID":"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd","Type":"ContainerStarted","Data":"8cada398515d4d43e78187077333b1e22a455203110e7c21b249c8597f2c03ea"}
Nov 03 18:20:27.313675 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:27.313566    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d" event={"ID":"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd","Type":"ContainerStarted","Data":"14809fc519d2913f70b175de42564e5fa0748fffe1982334f6588da1856db16f"}
Nov 03 18:20:27.551309 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:27.551211    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d" podStartSLOduration=2.551195914 podStartE2EDuration="2.551195914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:20:27.550225869 +0000 UTC m=+2807.009396856" watchObservedRunningTime="2026-11-03 18:20:27.551195914 +0000 UTC m=+2807.010366899"
Nov 03 18:20:28.034346 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.034310    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"]
Nov 03 18:20:28.321982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.321863    4493 generic.go:356] "Generic (PLEG): container finished" podID="eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" containerID="8cada398515d4d43e78187077333b1e22a455203110e7c21b249c8597f2c03ea" exitCode=0
Nov 03 18:20:28.322136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.321906    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d" event={"ID":"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd","Type":"ContainerDied","Data":"8cada398515d4d43e78187077333b1e22a455203110e7c21b249c8597f2c03ea"}
Nov 03 18:20:28.495986 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.495960    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:28.656153 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.656064    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-openshift-ca\") pod \"0448e549-eef3-4706-81ac-f26c44eca459\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") "
Nov 03 18:20:28.656322 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.656207    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-xks-ca\") pod \"0448e549-eef3-4706-81ac-f26c44eca459\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") "
Nov 03 18:20:28.656387 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.656351    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0448e549-eef3-4706-81ac-f26c44eca459-kube-api-access-btqd7\" (UniqueName: \"kubernetes.io/projected/0448e549-eef3-4706-81ac-f26c44eca459-kube-api-access-btqd7\") pod \"0448e549-eef3-4706-81ac-f26c44eca459\" (UID: \"0448e549-eef3-4706-81ac-f26c44eca459\") "
Nov 03 18:20:28.657695 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.657639    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-openshift-ca" pod "0448e549-eef3-4706-81ac-f26c44eca459" (UID: "0448e549-eef3-4706-81ac-f26c44eca459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:28.659315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.659287    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0448e549-eef3-4706-81ac-f26c44eca459-kube-api-access-btqd7" pod "0448e549-eef3-4706-81ac-f26c44eca459" (UID: "0448e549-eef3-4706-81ac-f26c44eca459"). InnerVolumeSpecName "kube-api-access-btqd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:28.660244 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.660219    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-xks-ca" pod "0448e549-eef3-4706-81ac-f26c44eca459" (UID: "0448e549-eef3-4706-81ac-f26c44eca459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:28.757393 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.757368    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:28.757393 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.757393    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqd7\" (UniqueName: \"kubernetes.io/projected/0448e549-eef3-4706-81ac-f26c44eca459-kube-api-access-btqd7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:28.757579 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:28.757403    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0448e549-eef3-4706-81ac-f26c44eca459-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:29.267316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.267286    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"]
Nov 03 18:20:29.269707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.267725    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0448e549-eef3-4706-81ac-f26c44eca459" containerName="revoke-keys"
Nov 03 18:20:29.269707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.267736    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0448e549-eef3-4706-81ac-f26c44eca459" containerName="revoke-keys"
Nov 03 18:20:29.269707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.267803    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0448e549-eef3-4706-81ac-f26c44eca459" containerName="revoke-keys"
Nov 03 18:20:29.270704 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.270686    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.331535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.331501    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"]
Nov 03 18:20:29.333593 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.333560    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs" event={"ID":"0448e549-eef3-4706-81ac-f26c44eca459","Type":"ContainerDied","Data":"410fda888d455a3273faca676aa8af519f3cd0ae6e1268ad6d1a18ed2aa6e1a4"}
Nov 03 18:20:29.333736 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.333618    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410fda888d455a3273faca676aa8af519f3cd0ae6e1268ad6d1a18ed2aa6e1a4"
Nov 03 18:20:29.334678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.334204    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"
Nov 03 18:20:29.364510 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.364476    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.364715 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.364551    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.364715 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.364697    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpbn\" (UniqueName: \"kubernetes.io/projected/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-kube-api-access-zmpbn\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.465390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.465364    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.465559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.465518    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpbn\" (UniqueName: \"kubernetes.io/projected/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-kube-api-access-zmpbn\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.465632 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.465570    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.466252 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.466183    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.467840 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.467820    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.468085 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.468071    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:29.529086 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.529026    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpbn\" (UniqueName: \"kubernetes.io/projected/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-kube-api-access-zmpbn\") pod \"maas-api-revoke-sub-ea9ffb86-q5846\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.566745 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.566719    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-openshift-ca\") pod \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") "
Nov 03 18:20:29.566893 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.566770    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-kube-api-access-wjk8q\" (UniqueName: \"kubernetes.io/projected/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-kube-api-access-wjk8q\") pod \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") "
Nov 03 18:20:29.566960 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.566925    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-xks-ca\") pod \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\" (UID: \"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd\") "
Nov 03 18:20:29.567049 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.567028    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-openshift-ca" pod "eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" (UID: "eebc30cd-e785-4a2a-88ed-1f6143bc0bdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:29.567322 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.567306    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:29.568889 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.568853    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-kube-api-access-wjk8q" pod "eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" (UID: "eebc30cd-e785-4a2a-88ed-1f6143bc0bdd"). InnerVolumeSpecName "kube-api-access-wjk8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:29.568973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.568922    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-xks-ca" pod "eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" (UID: "eebc30cd-e785-4a2a-88ed-1f6143bc0bdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:29.583077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.583056    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:29.669164 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.669130    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjk8q\" (UniqueName: \"kubernetes.io/projected/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-kube-api-access-wjk8q\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:29.669502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.669168    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:29.965555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:29.965523    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"]
Nov 03 18:20:29.966571 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:20:29.966545    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e66a96_1f1f_4e19_90eb_3f6c3b5e569a.slice/crio-4b649491d5164ba1042fa836c87d50c0d43de43a7d704e6b4ebbfdff822bbbce WatchSource:0}: Error finding container 4b649491d5164ba1042fa836c87d50c0d43de43a7d704e6b4ebbfdff822bbbce: Status 404 returned error can't find the container with id 4b649491d5164ba1042fa836c87d50c0d43de43a7d704e6b4ebbfdff822bbbce
Nov 03 18:20:30.340380 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:30.340344    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d" event={"ID":"eebc30cd-e785-4a2a-88ed-1f6143bc0bdd","Type":"ContainerDied","Data":"14809fc519d2913f70b175de42564e5fa0748fffe1982334f6588da1856db16f"}
Nov 03 18:20:30.340832 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:30.340356    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"
Nov 03 18:20:30.340832 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:30.340441    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14809fc519d2913f70b175de42564e5fa0748fffe1982334f6588da1856db16f"
Nov 03 18:20:30.342272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:30.342246    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846" event={"ID":"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a","Type":"ContainerStarted","Data":"e98bafb5db56d65f85315f07b1797328b92d0bf5634a148c8fc5980a3cd78649"}
Nov 03 18:20:30.342407 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:30.342299    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846" event={"ID":"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a","Type":"ContainerStarted","Data":"4b649491d5164ba1042fa836c87d50c0d43de43a7d704e6b4ebbfdff822bbbce"}
Nov 03 18:20:30.463215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:30.463167    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846" podStartSLOduration=1.46315369 podStartE2EDuration="1.46315369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:20:30.461941747 +0000 UTC m=+2809.921112734" watchObservedRunningTime="2026-11-03 18:20:30.46315369 +0000 UTC m=+2809.922324679"
Nov 03 18:20:31.349469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:31.349424    4493 generic.go:356] "Generic (PLEG): container finished" podID="63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" containerID="e98bafb5db56d65f85315f07b1797328b92d0bf5634a148c8fc5980a3cd78649" exitCode=0
Nov 03 18:20:31.349864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:31.349468    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846" event={"ID":"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a","Type":"ContainerDied","Data":"e98bafb5db56d65f85315f07b1797328b92d0bf5634a148c8fc5980a3cd78649"}
Nov 03 18:20:32.495365 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.495339    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:32.599005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.598976    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-kube-api-access-zmpbn\" (UniqueName: \"kubernetes.io/projected/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-kube-api-access-zmpbn\") pod \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") "
Nov 03 18:20:32.599178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.599021    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-openshift-ca\") pod \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") "
Nov 03 18:20:32.599178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.599062    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-xks-ca\") pod \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\" (UID: \"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a\") "
Nov 03 18:20:32.599459 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.599423    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-openshift-ca" pod "63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" (UID: "63e66a96-1f1f-4e19-90eb-3f6c3b5e569a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:32.601375 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.601344    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-kube-api-access-zmpbn" pod "63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" (UID: "63e66a96-1f1f-4e19-90eb-3f6c3b5e569a"). InnerVolumeSpecName "kube-api-access-zmpbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:32.601514 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.601387    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-xks-ca" pod "63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" (UID: "63e66a96-1f1f-4e19-90eb-3f6c3b5e569a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:32.700654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.700619    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmpbn\" (UniqueName: \"kubernetes.io/projected/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-kube-api-access-zmpbn\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:32.700654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.700650    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:32.700654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:32.700660    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:33.363051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:33.363017    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"
Nov 03 18:20:33.363232 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:33.363053    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846" event={"ID":"63e66a96-1f1f-4e19-90eb-3f6c3b5e569a","Type":"ContainerDied","Data":"4b649491d5164ba1042fa836c87d50c0d43de43a7d704e6b4ebbfdff822bbbce"}
Nov 03 18:20:33.363232 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:33.363103    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b649491d5164ba1042fa836c87d50c0d43de43a7d704e6b4ebbfdff822bbbce"
Nov 03 18:20:35.464211 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.464173    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"]
Nov 03 18:20:35.464683 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.464669    4493 cpu_manager.go:414] "containerMap: removing container" podUID="eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" containerName="revoke-keys"
Nov 03 18:20:35.464734 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.464685    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" containerName="revoke-keys"
Nov 03 18:20:35.464734 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.464711    4493 cpu_manager.go:414] "containerMap: removing container" podUID="63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" containerName="revoke-keys"
Nov 03 18:20:35.464734 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.464716    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" containerName="revoke-keys"
Nov 03 18:20:35.464834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.464793    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" containerName="revoke-keys"
Nov 03 18:20:35.464834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.464811    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" containerName="revoke-keys"
Nov 03 18:20:35.468071 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.468055    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.527856 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.527827    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"]
Nov 03 18:20:35.574916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.574856    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:20:35.574916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.574908    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:20:35.575125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.574934    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:20:35.575125 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.574985    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:20:35.630803 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.630764    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95pw9\" (UniqueName: \"kubernetes.io/projected/1b218d4b-55a7-40ad-8c98-bec70606d7d4-kube-api-access-95pw9\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.631006 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.630909    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.631063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.631015    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.741754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.741655    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.742416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.742382    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95pw9\" (UniqueName: \"kubernetes.io/projected/1b218d4b-55a7-40ad-8c98-bec70606d7d4-kube-api-access-95pw9\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.742564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.742497    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.743176 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.743149    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.745141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.745115    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:35.806861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:35.806824    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95pw9\" (UniqueName: \"kubernetes.io/projected/1b218d4b-55a7-40ad-8c98-bec70606d7d4-kube-api-access-95pw9\") pod \"maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:36.082442 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:36.082363    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:36.465074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:36.465042    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"]
Nov 03 18:20:36.466759 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:20:36.466733    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b218d4b_55a7_40ad_8c98_bec70606d7d4.slice/crio-2ee5ace29a4a4f242fa1fb40260ca6ed42353cabb5d9b07cb56c3c14b5e4c9e3 WatchSource:0}: Error finding container 2ee5ace29a4a4f242fa1fb40260ca6ed42353cabb5d9b07cb56c3c14b5e4c9e3: Status 404 returned error can't find the container with id 2ee5ace29a4a4f242fa1fb40260ca6ed42353cabb5d9b07cb56c3c14b5e4c9e3
Nov 03 18:20:36.992779 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:20:36.992691    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:20:37.388916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:37.388862    4493 generic.go:356] "Generic (PLEG): container finished" podID="1b218d4b-55a7-40ad-8c98-bec70606d7d4" containerID="28f1b5df2a1ccf813a4cbb970ee35a7b67264cd86cf8ea2efda50c550ad24e87" exitCode=0
Nov 03 18:20:37.389113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:37.388966    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd" event={"ID":"1b218d4b-55a7-40ad-8c98-bec70606d7d4","Type":"ContainerDied","Data":"28f1b5df2a1ccf813a4cbb970ee35a7b67264cd86cf8ea2efda50c550ad24e87"}
Nov 03 18:20:37.389113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:37.389059    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd" event={"ID":"1b218d4b-55a7-40ad-8c98-bec70606d7d4","Type":"ContainerStarted","Data":"2ee5ace29a4a4f242fa1fb40260ca6ed42353cabb5d9b07cb56c3c14b5e4c9e3"}
Nov 03 18:20:38.533016 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.532993    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:38.577755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.577724    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-openshift-ca\") pod \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") "
Nov 03 18:20:38.577949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.577796    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-xks-ca\") pod \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") "
Nov 03 18:20:38.577949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.577851    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b218d4b-55a7-40ad-8c98-bec70606d7d4-kube-api-access-95pw9\" (UniqueName: \"kubernetes.io/projected/1b218d4b-55a7-40ad-8c98-bec70606d7d4-kube-api-access-95pw9\") pod \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\" (UID: \"1b218d4b-55a7-40ad-8c98-bec70606d7d4\") "
Nov 03 18:20:38.578150 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.578122    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-openshift-ca" pod "1b218d4b-55a7-40ad-8c98-bec70606d7d4" (UID: "1b218d4b-55a7-40ad-8c98-bec70606d7d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:20:38.578325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.578306    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:38.580024 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.579995    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-xks-ca" pod "1b218d4b-55a7-40ad-8c98-bec70606d7d4" (UID: "1b218d4b-55a7-40ad-8c98-bec70606d7d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:38.580113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.580092    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b218d4b-55a7-40ad-8c98-bec70606d7d4-kube-api-access-95pw9" pod "1b218d4b-55a7-40ad-8c98-bec70606d7d4" (UID: "1b218d4b-55a7-40ad-8c98-bec70606d7d4"). InnerVolumeSpecName "kube-api-access-95pw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:38.680173 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.680092    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95pw9\" (UniqueName: \"kubernetes.io/projected/1b218d4b-55a7-40ad-8c98-bec70606d7d4-kube-api-access-95pw9\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:38.680173 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:38.680132    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b218d4b-55a7-40ad-8c98-bec70606d7d4-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:39.402331 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:39.402300    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"
Nov 03 18:20:39.402512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:39.402326    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd" event={"ID":"1b218d4b-55a7-40ad-8c98-bec70606d7d4","Type":"ContainerDied","Data":"2ee5ace29a4a4f242fa1fb40260ca6ed42353cabb5d9b07cb56c3c14b5e4c9e3"}
Nov 03 18:20:39.402512 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:39.402375    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee5ace29a4a4f242fa1fb40260ca6ed42353cabb5d9b07cb56c3c14b5e4c9e3"
Nov 03 18:20:39.875116 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:39.875075    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"]
Nov 03 18:20:39.883998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:39.883963    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"]
Nov 03 18:20:43.055469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.055427    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl" podUID="54fe4529-f34b-4db0-8355-dea8d7ce5aa0" containerName="main" containerID="cri-o://c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc" gracePeriod=45
Nov 03 18:20:43.117170 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.117138    4493 scope.go:122] "RemoveContainer" containerID="64daefbe97cfd817c7aa5e418b3041174a1761aa9524cca6ae08410525290671"
Nov 03 18:20:43.318034 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.317960    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:20:43.429104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.429067    4493 generic.go:356] "Generic (PLEG): container finished" podID="54fe4529-f34b-4db0-8355-dea8d7ce5aa0" containerID="c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc" exitCode=0
Nov 03 18:20:43.429325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.429113    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"
Nov 03 18:20:43.429325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.429161    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl" event={"ID":"54fe4529-f34b-4db0-8355-dea8d7ce5aa0","Type":"ContainerDied","Data":"c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc"}
Nov 03 18:20:43.429325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.429204    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl" event={"ID":"54fe4529-f34b-4db0-8355-dea8d7ce5aa0","Type":"ContainerDied","Data":"c6a7321bd10bc54fdc81ca2ae75ed254b3271131eb6ea75278c4037b2d5f467f"}
Nov 03 18:20:43.429325 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.429225    4493 scope.go:122] "RemoveContainer" containerID="c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc"
Nov 03 18:20:43.432617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.432594    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tls-certs\") pod \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") "
Nov 03 18:20:43.432783 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.432769    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-home\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-home\") pod \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") "
Nov 03 18:20:43.432854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.432809    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-kube-api-access-4rchb\" (UniqueName: \"kubernetes.io/projected/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-kube-api-access-4rchb\") pod \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") "
Nov 03 18:20:43.432854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.432829    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-model-cache\") pod \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") "
Nov 03 18:20:43.433020 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.432893    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-dshm\") pod \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") "
Nov 03 18:20:43.433020 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.432922    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tmp-dir\") pod \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\" (UID: \"54fe4529-f34b-4db0-8355-dea8d7ce5aa0\") "
Nov 03 18:20:43.433140 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.433113    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-model-cache" pod "54fe4529-f34b-4db0-8355-dea8d7ce5aa0" (UID: "54fe4529-f34b-4db0-8355-dea8d7ce5aa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:43.433140 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.433125    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-home" pod "54fe4529-f34b-4db0-8355-dea8d7ce5aa0" (UID: "54fe4529-f34b-4db0-8355-dea8d7ce5aa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:43.433497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.433461    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tmp-dir" pod "54fe4529-f34b-4db0-8355-dea8d7ce5aa0" (UID: "54fe4529-f34b-4db0-8355-dea8d7ce5aa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:43.433631 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.433547    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.433631 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.433569    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.435270 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.435245    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tls-certs" pod "54fe4529-f34b-4db0-8355-dea8d7ce5aa0" (UID: "54fe4529-f34b-4db0-8355-dea8d7ce5aa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:43.435552 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.435527    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-dshm" pod "54fe4529-f34b-4db0-8355-dea8d7ce5aa0" (UID: "54fe4529-f34b-4db0-8355-dea8d7ce5aa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:43.435790 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.435760    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-kube-api-access-4rchb" pod "54fe4529-f34b-4db0-8355-dea8d7ce5aa0" (UID: "54fe4529-f34b-4db0-8355-dea8d7ce5aa0"). InnerVolumeSpecName "kube-api-access-4rchb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:43.440243 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.439177    4493 scope.go:122] "RemoveContainer" containerID="c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc"
Nov 03 18:20:43.440243 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:20:43.439524    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc\": container with ID starting with c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc not found: ID does not exist" containerID="c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc"
Nov 03 18:20:43.440243 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.439560    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc"} err="failed to get container status \"c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc\": rpc error: code = NotFound desc = could not find container \"c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc\": container with ID starting with c15ebdee88e9a33b4708716cfcebb371acaa116cd4adf2f9192162361186b7fc not found: ID does not exist"
Nov 03 18:20:43.534596 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.534545    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rchb\" (UniqueName: \"kubernetes.io/projected/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-kube-api-access-4rchb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.534596 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.534589    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.534596 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.534603    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.534853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.534615    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fe4529-f34b-4db0-8355-dea8d7ce5aa0-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:43.857347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.857313    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"]
Nov 03 18:20:43.907748 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:43.907715    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ca6b33/test-model-ca6b33-kserve-7fc9cfb47-fd4kl"]
Nov 03 18:20:44.712381 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:44.712349    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fe4529-f34b-4db0-8355-dea8d7ce5aa0" path="/var/lib/kubelet/pods/54fe4529-f34b-4db0-8355-dea8d7ce5aa0/volumes"
Nov 03 18:20:47.108688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:47.108655    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"]
Nov 03 18:20:47.219097 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:47.219061    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"]
Nov 03 18:20:52.109850 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:52.109762    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" podUID="3b2fea08-041f-41d9-95be-811f63cc4434" containerName="payload-processing" containerID="cri-o://fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3" gracePeriod=25
Nov 03 18:20:52.219969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:52.219901    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" podUID="7786ff62-727a-4d61-98f6-e3f1071c97f7" containerName="payload-pre-processing" containerID="cri-o://5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479" gracePeriod=25
Nov 03 18:20:53.057095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:53.057059    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"]
Nov 03 18:20:53.107902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:53.107859    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f1b8c74-m8vmv"]
Nov 03 18:20:53.848678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:53.848639    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" podUID="7786ff62-727a-4d61-98f6-e3f1071c97f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:20:53.848678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:53.848677    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" podUID="3b2fea08-041f-41d9-95be-811f63cc4434" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:20:53.936789 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:53.936755    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"]
Nov 03 18:20:53.957690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:53.957658    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-f7d463-j6kjh"]
Nov 03 18:20:54.057503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.057465    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"]
Nov 03 18:20:54.061981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.061953    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4b18fbf-drq9m"]
Nov 03 18:20:54.711559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.711525    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8175b8ed-761b-4a02-997d-fc9c2871640e" path="/var/lib/kubelet/pods/8175b8ed-761b-4a02-997d-fc9c2871640e/volumes"
Nov 03 18:20:54.712004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.711989    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76421f5-61b3-460a-b26b-9f980cfdc9aa" path="/var/lib/kubelet/pods/a76421f5-61b3-460a-b26b-9f980cfdc9aa/volumes"
Nov 03 18:20:54.712357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.712344    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa034827-8abd-4269-b345-e4075b230588" path="/var/lib/kubelet/pods/fa034827-8abd-4269-b345-e4075b230588/volumes"
Nov 03 18:20:54.896345 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.896299    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" containerID="cri-o://21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be" gracePeriod=45
Nov 03 18:20:54.903855 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.903820    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main" containerID="cri-o://32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd" gracePeriod=45
Nov 03 18:20:54.908395 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:54.908365    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused"
Nov 03 18:20:55.168862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.168836    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 18:20:55.204630 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.204604    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 18:20:55.269623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269546    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-home\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-home\") pod \"c34f5790-bf7f-432c-98dc-d2c206f1f307\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") "
Nov 03 18:20:55.269623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269618    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-tmp-dir\") pod \"c34f5790-bf7f-432c-98dc-d2c206f1f307\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") "
Nov 03 18:20:55.269888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269641    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-dshm\") pod \"c34f5790-bf7f-432c-98dc-d2c206f1f307\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") "
Nov 03 18:20:55.269888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269676    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34f5790-bf7f-432c-98dc-d2c206f1f307-kube-api-access-zh4fb\" (UniqueName: \"kubernetes.io/projected/c34f5790-bf7f-432c-98dc-d2c206f1f307-kube-api-access-zh4fb\") pod \"c34f5790-bf7f-432c-98dc-d2c206f1f307\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") "
Nov 03 18:20:55.269888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269736    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-model-cache\") pod \"c34f5790-bf7f-432c-98dc-d2c206f1f307\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") "
Nov 03 18:20:55.269888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269758    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-home" pod "c34f5790-bf7f-432c-98dc-d2c206f1f307" (UID: "c34f5790-bf7f-432c-98dc-d2c206f1f307"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.270112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269908    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34f5790-bf7f-432c-98dc-d2c206f1f307-tls-certs\" (UniqueName: \"kubernetes.io/secret/c34f5790-bf7f-432c-98dc-d2c206f1f307-tls-certs\") pod \"c34f5790-bf7f-432c-98dc-d2c206f1f307\" (UID: \"c34f5790-bf7f-432c-98dc-d2c206f1f307\") "
Nov 03 18:20:55.270112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.269928    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-tmp-dir" pod "c34f5790-bf7f-432c-98dc-d2c206f1f307" (UID: "c34f5790-bf7f-432c-98dc-d2c206f1f307"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.270112 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.270032    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-model-cache" pod "c34f5790-bf7f-432c-98dc-d2c206f1f307" (UID: "c34f5790-bf7f-432c-98dc-d2c206f1f307"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.270404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.270376    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.270404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.270401    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.270515 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.270415    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.272120 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.272093    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-dshm" pod "c34f5790-bf7f-432c-98dc-d2c206f1f307" (UID: "c34f5790-bf7f-432c-98dc-d2c206f1f307"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.272235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.272134    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34f5790-bf7f-432c-98dc-d2c206f1f307-tls-certs" pod "c34f5790-bf7f-432c-98dc-d2c206f1f307" (UID: "c34f5790-bf7f-432c-98dc-d2c206f1f307"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:55.272235 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.272143    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34f5790-bf7f-432c-98dc-d2c206f1f307-kube-api-access-zh4fb" pod "c34f5790-bf7f-432c-98dc-d2c206f1f307" (UID: "c34f5790-bf7f-432c-98dc-d2c206f1f307"). InnerVolumeSpecName "kube-api-access-zh4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:55.371393 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371347    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-model-cache\") pod \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") "
Nov 03 18:20:55.371598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371438    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-dshm\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-dshm\") pod \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") "
Nov 03 18:20:55.371598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371522    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tmp-dir\") pod \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") "
Nov 03 18:20:55.371598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371561    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-kube-api-access-4l26z\" (UniqueName: \"kubernetes.io/projected/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-kube-api-access-4l26z\") pod \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") "
Nov 03 18:20:55.371770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371622    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-model-cache" pod "419b56c2-9ba5-4f34-95bc-a2f45ae3d256" (UID: "419b56c2-9ba5-4f34-95bc-a2f45ae3d256"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.371770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371631    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tls-certs\" (UniqueName: \"kubernetes.io/secret/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tls-certs\") pod \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") "
Nov 03 18:20:55.371770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371702    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-home\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-home\") pod \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\" (UID: \"419b56c2-9ba5-4f34-95bc-a2f45ae3d256\") "
Nov 03 18:20:55.371972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.371891    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tmp-dir" pod "419b56c2-9ba5-4f34-95bc-a2f45ae3d256" (UID: "419b56c2-9ba5-4f34-95bc-a2f45ae3d256"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.372440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.372184    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-home" pod "419b56c2-9ba5-4f34-95bc-a2f45ae3d256" (UID: "419b56c2-9ba5-4f34-95bc-a2f45ae3d256"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.372440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.372325    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.372440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.372346    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34f5790-bf7f-432c-98dc-d2c206f1f307-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.372440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.372360    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.372440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.372376    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34f5790-bf7f-432c-98dc-d2c206f1f307-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.372440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.372390    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh4fb\" (UniqueName: \"kubernetes.io/projected/c34f5790-bf7f-432c-98dc-d2c206f1f307-kube-api-access-zh4fb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.372440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.372404    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.373920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.373895    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tls-certs" pod "419b56c2-9ba5-4f34-95bc-a2f45ae3d256" (UID: "419b56c2-9ba5-4f34-95bc-a2f45ae3d256"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:20:55.373920 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.373903    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-dshm" pod "419b56c2-9ba5-4f34-95bc-a2f45ae3d256" (UID: "419b56c2-9ba5-4f34-95bc-a2f45ae3d256"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:20:55.374599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.374582    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-kube-api-access-4l26z" pod "419b56c2-9ba5-4f34-95bc-a2f45ae3d256" (UID: "419b56c2-9ba5-4f34-95bc-a2f45ae3d256"). InnerVolumeSpecName "kube-api-access-4l26z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:20:55.473374 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.473341    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.473374 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.473372    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l26z\" (UniqueName: \"kubernetes.io/projected/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-kube-api-access-4l26z\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.473580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.473382    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/419b56c2-9ba5-4f34-95bc-a2f45ae3d256-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:20:55.498444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.498414    4493 generic.go:356] "Generic (PLEG): container finished" podID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerID="21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be" exitCode=0
Nov 03 18:20:55.498622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.498459    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"
Nov 03 18:20:55.498622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.498491    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" event={"ID":"c34f5790-bf7f-432c-98dc-d2c206f1f307","Type":"ContainerDied","Data":"21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be"}
Nov 03 18:20:55.498622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.498541    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk" event={"ID":"c34f5790-bf7f-432c-98dc-d2c206f1f307","Type":"ContainerDied","Data":"b9b02165a834cc184f7233a9ccf3989529b3c6ce04d9847164b614fbd4f315b8"}
Nov 03 18:20:55.498622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.498564    4493 scope.go:122] "RemoveContainer" containerID="21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be"
Nov 03 18:20:55.500074 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.500052    4493 generic.go:356] "Generic (PLEG): container finished" podID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerID="32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd" exitCode=0
Nov 03 18:20:55.500217 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.500097    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" event={"ID":"419b56c2-9ba5-4f34-95bc-a2f45ae3d256","Type":"ContainerDied","Data":"32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd"}
Nov 03 18:20:55.500217 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.500125    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp" event={"ID":"419b56c2-9ba5-4f34-95bc-a2f45ae3d256","Type":"ContainerDied","Data":"27a6bb11f3bf53f9b81a99c9b8cd09da543e8c77410bb7a4f7c506b06fd93699"}
Nov 03 18:20:55.500342 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.500251    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"
Nov 03 18:20:55.509182 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.509165    4493 scope.go:122] "RemoveContainer" containerID="21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be"
Nov 03 18:20:55.509488 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:20:55.509455    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be\": container with ID starting with 21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be not found: ID does not exist" containerID="21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be"
Nov 03 18:20:55.509535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.509502    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be"} err="failed to get container status \"21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be\": rpc error: code = NotFound desc = could not find container \"21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be\": container with ID starting with 21e1718f712a7a6ae38c3954c2c1634d019b8dd0344865a7ffd34599c8bec6be not found: ID does not exist"
Nov 03 18:20:55.509535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.509528    4493 scope.go:122] "RemoveContainer" containerID="32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd"
Nov 03 18:20:55.517586 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.517570    4493 scope.go:122] "RemoveContainer" containerID="32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd"
Nov 03 18:20:55.517866 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:20:55.517848    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd\": container with ID starting with 32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd not found: ID does not exist" containerID="32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd"
Nov 03 18:20:55.517966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.517944    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd"} err="failed to get container status \"32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd\": rpc error: code = NotFound desc = could not find container \"32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd\": container with ID starting with 32d9ff6adf61ebc10283cf69647468232b5e475fe276f1cb7c8af883f5400bbd not found: ID does not exist"
Nov 03 18:20:55.670297 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.670222    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"]
Nov 03 18:20:55.678762 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.678724    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/simulator-w4-73a606-kserve-86cdd6546-cz6hp"]
Nov 03 18:20:55.792654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.792625    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"]
Nov 03 18:20:55.794803 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:55.794782    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/premium-w4-73a606-kserve-75b8ff487-qj8jk"]
Nov 03 18:20:56.711703 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:56.711669    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" path="/var/lib/kubelet/pods/419b56c2-9ba5-4f34-95bc-a2f45ae3d256/volumes"
Nov 03 18:20:56.712115 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:56.712100    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" path="/var/lib/kubelet/pods/c34f5790-bf7f-432c-98dc-d2c206f1f307/volumes"
Nov 03 18:20:59.059258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:59.059221    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"]
Nov 03 18:20:59.111544 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:59.111505    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"]
Nov 03 18:20:59.114761 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:59.114735    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634be419-c5xnc"]
Nov 03 18:20:59.117658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:20:59.117634    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0346dc1-vs2br"]
Nov 03 18:21:00.712155 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:00.712123    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ae6c75-4276-4216-b218-03a575bc1e7e" path="/var/lib/kubelet/pods/17ae6c75-4276-4216-b218-03a575bc1e7e/volumes"
Nov 03 18:21:00.712534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:00.712520    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc872fd-ab89-4113-8e61-46d9e63902c7" path="/var/lib/kubelet/pods/3fc872fd-ab89-4113-8e61-46d9e63902c7/volumes"
Nov 03 18:21:02.258275 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.258232    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"]
Nov 03 18:21:02.852947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.852909    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"]
Nov 03 18:21:02.853305 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853292    4493 cpu_manager.go:414] "containerMap: removing container" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main"
Nov 03 18:21:02.853354 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853306    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main"
Nov 03 18:21:02.853354 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853323    4493 cpu_manager.go:414] "containerMap: removing container" podUID="54fe4529-f34b-4db0-8355-dea8d7ce5aa0" containerName="main"
Nov 03 18:21:02.853354 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853329    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fe4529-f34b-4db0-8355-dea8d7ce5aa0" containerName="main"
Nov 03 18:21:02.853354 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853345    4493 cpu_manager.go:414] "containerMap: removing container" podUID="1b218d4b-55a7-40ad-8c98-bec70606d7d4" containerName="revoke-keys"
Nov 03 18:21:02.853354 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853350    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b218d4b-55a7-40ad-8c98-bec70606d7d4" containerName="revoke-keys"
Nov 03 18:21:02.853529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853357    4493 cpu_manager.go:414] "containerMap: removing container" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main"
Nov 03 18:21:02.853529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853363    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main"
Nov 03 18:21:02.853529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853437    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34f5790-bf7f-432c-98dc-d2c206f1f307" containerName="main"
Nov 03 18:21:02.853529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853449    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fe4529-f34b-4db0-8355-dea8d7ce5aa0" containerName="main"
Nov 03 18:21:02.853529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853456    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b218d4b-55a7-40ad-8c98-bec70606d7d4" containerName="revoke-keys"
Nov 03 18:21:02.853529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.853463    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="419b56c2-9ba5-4f34-95bc-a2f45ae3d256" containerName="main"
Nov 03 18:21:02.858179 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.858159    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:02.913126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.913081    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:21:02.914380 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.914357    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:21:02.916063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.916039    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"]
Nov 03 18:21:02.944070 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.944042    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:02.944243 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.944220    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzk8\" (UniqueName: \"kubernetes.io/projected/e38b02b7-f02f-4407-a97b-5afcc760d0b6-kube-api-access-7pzk8\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:02.944308 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.944290    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:02.963547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.963515    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:21:02.977333 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:02.977307    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:21:03.045334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.045298    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:03.045528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.045376    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzk8\" (UniqueName: \"kubernetes.io/projected/e38b02b7-f02f-4407-a97b-5afcc760d0b6-kube-api-access-7pzk8\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:03.045528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.045418    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:03.046144 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.046119    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:03.047837 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.047817    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:03.111507 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.111443    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzk8\" (UniqueName: \"kubernetes.io/projected/e38b02b7-f02f-4407-a97b-5afcc760d0b6-kube-api-access-7pzk8\") pod \"maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:03.171406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.171374    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:03.418740 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.418665    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"]
Nov 03 18:21:03.423448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.423426    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.451399 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.451373    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cslqb\" (UniqueName: \"kubernetes.io/projected/7eb97c6b-8697-4a9b-af7a-f5d760e26543-kube-api-access-cslqb\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.451526 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.451420    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.451588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.451569    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.483558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.483530    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"]
Nov 03 18:21:03.549743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.549694    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"]
Nov 03 18:21:03.550050 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.549998    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy" containerID="cri-o://042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a" gracePeriod=30
Nov 03 18:21:03.552622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.552589    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cslqb\" (UniqueName: \"kubernetes.io/projected/7eb97c6b-8697-4a9b-af7a-f5d760e26543-kube-api-access-cslqb\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.552758 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.552658    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.552758 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.552730    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.553553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.553501    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.555967 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.555935    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.597227 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.597181    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"]
Nov 03 18:21:03.600659 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:21:03.600630    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38b02b7_f02f_4407_a97b_5afcc760d0b6.slice/crio-1b947cc4070e3a49ca58650da6bc46dda3748b21bc23232048ada20b95c164ad WatchSource:0}: Error finding container 1b947cc4070e3a49ca58650da6bc46dda3748b21bc23232048ada20b95c164ad: Status 404 returned error can't find the container with id 1b947cc4070e3a49ca58650da6bc46dda3748b21bc23232048ada20b95c164ad
Nov 03 18:21:03.617151 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.617122    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cslqb\" (UniqueName: \"kubernetes.io/projected/7eb97c6b-8697-4a9b-af7a-f5d760e26543-kube-api-access-cslqb\") pod \"maas-api-revoke-sub-19710e6d-7zrcj\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.749801 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.738627    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:03.849205 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.849170    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" podUID="3b2fea08-041f-41d9-95be-811f63cc4434" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:21:03.849404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:03.849244    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" podUID="7786ff62-727a-4d61-98f6-e3f1071c97f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:21:04.183481 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:04.183454    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"]
Nov 03 18:21:04.185224 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:21:04.185192    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb97c6b_8697_4a9b_af7a_f5d760e26543.slice/crio-14e487eadbeab89b6cab58bfc12bef8e2d66837c7a4216ce74019b1ff88a20c0 WatchSource:0}: Error finding container 14e487eadbeab89b6cab58bfc12bef8e2d66837c7a4216ce74019b1ff88a20c0: Status 404 returned error can't find the container with id 14e487eadbeab89b6cab58bfc12bef8e2d66837c7a4216ce74019b1ff88a20c0
Nov 03 18:21:04.558817 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:04.558782    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj" event={"ID":"7eb97c6b-8697-4a9b-af7a-f5d760e26543","Type":"ContainerStarted","Data":"bbd5304ec5d8e804df5d97003ac765069907da801d8b775068278af01694c109"}
Nov 03 18:21:04.559267 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:04.558837    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj" event={"ID":"7eb97c6b-8697-4a9b-af7a-f5d760e26543","Type":"ContainerStarted","Data":"14e487eadbeab89b6cab58bfc12bef8e2d66837c7a4216ce74019b1ff88a20c0"}
Nov 03 18:21:04.560465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:04.560439    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf" event={"ID":"e38b02b7-f02f-4407-a97b-5afcc760d0b6","Type":"ContainerStarted","Data":"6f75c0aaa3b1a435a790eed2e4b61ea881650caa9672bf1c0fe0804f8ac81740"}
Nov 03 18:21:04.560589 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:04.560479    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf" event={"ID":"e38b02b7-f02f-4407-a97b-5afcc760d0b6","Type":"ContainerStarted","Data":"1b947cc4070e3a49ca58650da6bc46dda3748b21bc23232048ada20b95c164ad"}
Nov 03 18:21:04.680968 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:04.680917    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj" podStartSLOduration=1.680902946 podStartE2EDuration="1.680902946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:04.678681015 +0000 UTC m=+2844.137852000" watchObservedRunningTime="2026-11-03 18:21:04.680902946 +0000 UTC m=+2844.140073931"
Nov 03 18:21:04.798898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:04.798836    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf" podStartSLOduration=2.798821966 podStartE2EDuration="2.798821966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:04.797197852 +0000 UTC m=+2844.256368842" watchObservedRunningTime="2026-11-03 18:21:04.798821966 +0000 UTC m=+2844.257992978"
Nov 03 18:21:05.567044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:05.567008    4493 generic.go:356] "Generic (PLEG): container finished" podID="7eb97c6b-8697-4a9b-af7a-f5d760e26543" containerID="bbd5304ec5d8e804df5d97003ac765069907da801d8b775068278af01694c109" exitCode=0
Nov 03 18:21:05.567444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:05.567054    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj" event={"ID":"7eb97c6b-8697-4a9b-af7a-f5d760e26543","Type":"ContainerDied","Data":"bbd5304ec5d8e804df5d97003ac765069907da801d8b775068278af01694c109"}
Nov 03 18:21:05.568633 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:05.568607    4493 generic.go:356] "Generic (PLEG): container finished" podID="e38b02b7-f02f-4407-a97b-5afcc760d0b6" containerID="6f75c0aaa3b1a435a790eed2e4b61ea881650caa9672bf1c0fe0804f8ac81740" exitCode=0
Nov 03 18:21:05.568714 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:05.568691    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf" event={"ID":"e38b02b7-f02f-4407-a97b-5afcc760d0b6","Type":"ContainerDied","Data":"6f75c0aaa3b1a435a790eed2e4b61ea881650caa9672bf1c0fe0804f8ac81740"}
Nov 03 18:21:06.756004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.755976    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:06.759042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.759024    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:06.787585 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.787554    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-xks-ca\") pod \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") "
Nov 03 18:21:06.787749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.787601    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38b02b7-f02f-4407-a97b-5afcc760d0b6-kube-api-access-7pzk8\" (UniqueName: \"kubernetes.io/projected/e38b02b7-f02f-4407-a97b-5afcc760d0b6-kube-api-access-7pzk8\") pod \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") "
Nov 03 18:21:06.787749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.787639    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-openshift-ca\") pod \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") "
Nov 03 18:21:06.787749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.787693    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-openshift-ca\") pod \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\" (UID: \"e38b02b7-f02f-4407-a97b-5afcc760d0b6\") "
Nov 03 18:21:06.787749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.787720    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-xks-ca\") pod \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") "
Nov 03 18:21:06.788084 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.787767    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb97c6b-8697-4a9b-af7a-f5d760e26543-kube-api-access-cslqb\" (UniqueName: \"kubernetes.io/projected/7eb97c6b-8697-4a9b-af7a-f5d760e26543-kube-api-access-cslqb\") pod \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\" (UID: \"7eb97c6b-8697-4a9b-af7a-f5d760e26543\") "
Nov 03 18:21:06.788141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.788097    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-openshift-ca" pod "e38b02b7-f02f-4407-a97b-5afcc760d0b6" (UID: "e38b02b7-f02f-4407-a97b-5afcc760d0b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:06.788141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.788089    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-openshift-ca" pod "7eb97c6b-8697-4a9b-af7a-f5d760e26543" (UID: "7eb97c6b-8697-4a9b-af7a-f5d760e26543"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:06.789996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.789969    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-xks-ca" pod "e38b02b7-f02f-4407-a97b-5afcc760d0b6" (UID: "e38b02b7-f02f-4407-a97b-5afcc760d0b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:21:06.790121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.790058    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-xks-ca" pod "7eb97c6b-8697-4a9b-af7a-f5d760e26543" (UID: "7eb97c6b-8697-4a9b-af7a-f5d760e26543"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:21:06.790121 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.790107    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38b02b7-f02f-4407-a97b-5afcc760d0b6-kube-api-access-7pzk8" pod "e38b02b7-f02f-4407-a97b-5afcc760d0b6" (UID: "e38b02b7-f02f-4407-a97b-5afcc760d0b6"). InnerVolumeSpecName "kube-api-access-7pzk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:06.790199 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.790118    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb97c6b-8697-4a9b-af7a-f5d760e26543-kube-api-access-cslqb" pod "7eb97c6b-8697-4a9b-af7a-f5d760e26543" (UID: "7eb97c6b-8697-4a9b-af7a-f5d760e26543"). InnerVolumeSpecName "kube-api-access-cslqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:06.889488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.889403    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.889488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.889452    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.889488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.889469    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cslqb\" (UniqueName: \"kubernetes.io/projected/7eb97c6b-8697-4a9b-af7a-f5d760e26543-kube-api-access-cslqb\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.889488 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.889483    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b02b7-f02f-4407-a97b-5afcc760d0b6-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.889721 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.889496    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pzk8\" (UniqueName: \"kubernetes.io/projected/e38b02b7-f02f-4407-a97b-5afcc760d0b6-kube-api-access-7pzk8\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.889721 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:06.889510    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb97c6b-8697-4a9b-af7a-f5d760e26543-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:06.992774 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:21:06.992728    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:21:07.581588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:07.581556    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"
Nov 03 18:21:07.581785 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:07.581588    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj" event={"ID":"7eb97c6b-8697-4a9b-af7a-f5d760e26543","Type":"ContainerDied","Data":"14e487eadbeab89b6cab58bfc12bef8e2d66837c7a4216ce74019b1ff88a20c0"}
Nov 03 18:21:07.581785 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:07.581631    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14e487eadbeab89b6cab58bfc12bef8e2d66837c7a4216ce74019b1ff88a20c0"
Nov 03 18:21:07.583046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:07.583023    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"
Nov 03 18:21:07.583171 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:07.583063    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf" event={"ID":"e38b02b7-f02f-4407-a97b-5afcc760d0b6","Type":"ContainerDied","Data":"1b947cc4070e3a49ca58650da6bc46dda3748b21bc23232048ada20b95c164ad"}
Nov 03 18:21:07.583171 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:07.583109    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b947cc4070e3a49ca58650da6bc46dda3748b21bc23232048ada20b95c164ad"
Nov 03 18:21:08.810258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.810232    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 18:21:08.913017 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.912947    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-envoy\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913017 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.912986    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-certs\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913034    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-credential-socket\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913077    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-podinfo\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913111    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-kube-api-access-p6bsw\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-kube-api-access-p6bsw\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913167    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istiod-ca-cert\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913194    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-socket\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913300    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-token\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-token\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913311    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-certs" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:08.913553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913371    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-data\") pod \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\" (UID: \"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2\") "
Nov 03 18:21:08.913553 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913368    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-credential-socket" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:08.913777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913554    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-socket" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:08.913777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913610    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istiod-ca-cert" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:08.913777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913747    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-data" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:08.914052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.913989    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:08.914052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.914016    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:08.914052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.914034    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:08.914052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.914048    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:08.914052 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.914064    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:08.915451 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.915425    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-envoy" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:08.915781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.915762    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-token" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:08.915953 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.915923    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-kube-api-access-p6bsw" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "kube-api-access-p6bsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:08.916084 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:08.915972    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-podinfo" pod "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" (UID: "fcac96ab-c8c5-4b4a-b5e6-6575bec12de2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:21:09.015478 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.015432    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:09.015478 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.015472    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:09.015724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.015488    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6bsw\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-kube-api-access-p6bsw\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:09.015724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.015504    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:09.595989 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.595952    4493 generic.go:356] "Generic (PLEG): container finished" podID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerID="042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a" exitCode=0
Nov 03 18:21:09.596172 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.596002    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"
Nov 03 18:21:09.596172 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.596034    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" event={"ID":"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2","Type":"ContainerDied","Data":"042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a"}
Nov 03 18:21:09.596172 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.596091    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx" event={"ID":"fcac96ab-c8c5-4b4a-b5e6-6575bec12de2","Type":"ContainerDied","Data":"bf67ae29a51493f1386ea1d73afae725f07e66cb741f51708e83dc3035dd3311"}
Nov 03 18:21:09.596172 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.596120    4493 scope.go:122] "RemoveContainer" containerID="042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a"
Nov 03 18:21:09.605905 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.605860    4493 scope.go:122] "RemoveContainer" containerID="042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a"
Nov 03 18:21:09.606180 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:21:09.606162    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a\": container with ID starting with 042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a not found: ID does not exist" containerID="042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a"
Nov 03 18:21:09.606233 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.606191    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a"} err="failed to get container status \"042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a\": rpc error: code = NotFound desc = could not find container \"042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a\": container with ID starting with 042a144caa4ce8f3dd42f0588fd10cac4dddaa913af88604c966ac72109ccb9a not found: ID does not exist"
Nov 03 18:21:09.820157 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.820125    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"]
Nov 03 18:21:09.870755 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:09.870690    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-pjmlx"]
Nov 03 18:21:10.712188 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:10.712157    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" path="/var/lib/kubelet/pods/fcac96ab-c8c5-4b4a-b5e6-6575bec12de2/volumes"
Nov 03 18:21:12.058745 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.058714    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"]
Nov 03 18:21:12.108796 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.108766    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b5a56c1-qc7s7"]
Nov 03 18:21:12.378823 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.378794    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 18:21:12.449810 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.449783    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2fea08-041f-41d9-95be-811f63cc4434-kube-api-access-5k7ft\" (UniqueName: \"kubernetes.io/projected/3b2fea08-041f-41d9-95be-811f63cc4434-kube-api-access-5k7ft\") pod \"3b2fea08-041f-41d9-95be-811f63cc4434\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") "
Nov 03 18:21:12.450011 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.449964    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2fea08-041f-41d9-95be-811f63cc4434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2fea08-041f-41d9-95be-811f63cc4434-plugins-config-volume\") pod \"3b2fea08-041f-41d9-95be-811f63cc4434\" (UID: \"3b2fea08-041f-41d9-95be-811f63cc4434\") "
Nov 03 18:21:12.450397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.450369    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2fea08-041f-41d9-95be-811f63cc4434-plugins-config-volume" pod "3b2fea08-041f-41d9-95be-811f63cc4434" (UID: "3b2fea08-041f-41d9-95be-811f63cc4434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:12.452129 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.452105    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2fea08-041f-41d9-95be-811f63cc4434-kube-api-access-5k7ft" pod "3b2fea08-041f-41d9-95be-811f63cc4434" (UID: "3b2fea08-041f-41d9-95be-811f63cc4434"). InnerVolumeSpecName "kube-api-access-5k7ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:12.467452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.467429    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 18:21:12.551500 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.551464    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7786ff62-727a-4d61-98f6-e3f1071c97f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7786ff62-727a-4d61-98f6-e3f1071c97f7-plugins-config-volume\") pod \"7786ff62-727a-4d61-98f6-e3f1071c97f7\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") "
Nov 03 18:21:12.551691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.551518    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7786ff62-727a-4d61-98f6-e3f1071c97f7-kube-api-access-qm7kr\" (UniqueName: \"kubernetes.io/projected/7786ff62-727a-4d61-98f6-e3f1071c97f7-kube-api-access-qm7kr\") pod \"7786ff62-727a-4d61-98f6-e3f1071c97f7\" (UID: \"7786ff62-727a-4d61-98f6-e3f1071c97f7\") "
Nov 03 18:21:12.551771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.551751    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2fea08-041f-41d9-95be-811f63cc4434-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:12.551771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.551762    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7ft\" (UniqueName: \"kubernetes.io/projected/3b2fea08-041f-41d9-95be-811f63cc4434-kube-api-access-5k7ft\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:12.551923 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.551817    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7786ff62-727a-4d61-98f6-e3f1071c97f7-plugins-config-volume" pod "7786ff62-727a-4d61-98f6-e3f1071c97f7" (UID: "7786ff62-727a-4d61-98f6-e3f1071c97f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:12.553902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.553860    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7786ff62-727a-4d61-98f6-e3f1071c97f7-kube-api-access-qm7kr" pod "7786ff62-727a-4d61-98f6-e3f1071c97f7" (UID: "7786ff62-727a-4d61-98f6-e3f1071c97f7"). InnerVolumeSpecName "kube-api-access-qm7kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:12.615536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.615464    4493 generic.go:356] "Generic (PLEG): container finished" podID="7786ff62-727a-4d61-98f6-e3f1071c97f7" containerID="5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479" exitCode=0
Nov 03 18:21:12.615692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.615575    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"
Nov 03 18:21:12.615692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.615607    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" event={"ID":"7786ff62-727a-4d61-98f6-e3f1071c97f7","Type":"ContainerDied","Data":"5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479"}
Nov 03 18:21:12.615692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.615672    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4" event={"ID":"7786ff62-727a-4d61-98f6-e3f1071c97f7","Type":"ContainerDied","Data":"e48555f6d491ed431fafb689aaaf625133cf3740fdefad5cb16f514d0ce474dd"}
Nov 03 18:21:12.615856 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.615695    4493 scope.go:122] "RemoveContainer" containerID="5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479"
Nov 03 18:21:12.617029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.617012    4493 generic.go:356] "Generic (PLEG): container finished" podID="3b2fea08-041f-41d9-95be-811f63cc4434" containerID="fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3" exitCode=0
Nov 03 18:21:12.617168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.617062    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"
Nov 03 18:21:12.617168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.617127    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" event={"ID":"3b2fea08-041f-41d9-95be-811f63cc4434","Type":"ContainerDied","Data":"fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3"}
Nov 03 18:21:12.617272 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.617177    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6" event={"ID":"3b2fea08-041f-41d9-95be-811f63cc4434","Type":"ContainerDied","Data":"df7ac362983801b40579e891675fa5dbf3f4f07a521338cea98b13f52e8d10e2"}
Nov 03 18:21:12.627683 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.627659    4493 scope.go:122] "RemoveContainer" containerID="5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479"
Nov 03 18:21:12.628097 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:21:12.628074    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479\": container with ID starting with 5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479 not found: ID does not exist" containerID="5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479"
Nov 03 18:21:12.628183 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.628104    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479"} err="failed to get container status \"5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479\": rpc error: code = NotFound desc = could not find container \"5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479\": container with ID starting with 5bed0afa59a44f9ccda040dfc33ad20673fcb9c8a4ba73cf5f0b467870b0e479 not found: ID does not exist"
Nov 03 18:21:12.628183 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.628121    4493 scope.go:122] "RemoveContainer" containerID="fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3"
Nov 03 18:21:12.638595 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.638576    4493 scope.go:122] "RemoveContainer" containerID="fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3"
Nov 03 18:21:12.638902 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:21:12.638866    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3\": container with ID starting with fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3 not found: ID does not exist" containerID="fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3"
Nov 03 18:21:12.638951 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.638912    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3"} err="failed to get container status \"fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3\": rpc error: code = NotFound desc = could not find container \"fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3\": container with ID starting with fc0522575813b78e3b7f5741958299394ea5b90186a1ea04017a8efa0a3785f3 not found: ID does not exist"
Nov 03 18:21:12.652680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.652646    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7786ff62-727a-4d61-98f6-e3f1071c97f7-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:12.652786 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.652684    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7kr\" (UniqueName: \"kubernetes.io/projected/7786ff62-727a-4d61-98f6-e3f1071c97f7-kube-api-access-qm7kr\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:12.711400 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.711371    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3" path="/var/lib/kubelet/pods/ed80f6ad-e21d-41c4-9afb-6c4bcd9b31e3/volumes"
Nov 03 18:21:12.843029 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.842996    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"]
Nov 03 18:21:12.848972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.848939    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-73a606-66b8b7bd99-7qhg4"]
Nov 03 18:21:12.962105 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.962029    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"]
Nov 03 18:21:12.969617 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:12.969588    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-73a606-76b9f45457-8lxh6"]
Nov 03 18:21:14.713167 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:14.713128    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2fea08-041f-41d9-95be-811f63cc4434" path="/var/lib/kubelet/pods/3b2fea08-041f-41d9-95be-811f63cc4434/volumes"
Nov 03 18:21:14.713700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:14.713679    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7786ff62-727a-4d61-98f6-e3f1071c97f7" path="/var/lib/kubelet/pods/7786ff62-727a-4d61-98f6-e3f1071c97f7/volumes"
Nov 03 18:21:17.278448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.278384    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp" podUID="829de85e-943e-4d8a-a2c4-d517bad23734" containerName="main" containerID="cri-o://40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d" gracePeriod=45
Nov 03 18:21:17.537926 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.537839    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:21:17.604772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.604734    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829de85e-943e-4d8a-a2c4-d517bad23734-tls-certs\" (UniqueName: \"kubernetes.io/secret/829de85e-943e-4d8a-a2c4-d517bad23734-tls-certs\") pod \"829de85e-943e-4d8a-a2c4-d517bad23734\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") "
Nov 03 18:21:17.604987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.604789    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-model-cache\") pod \"829de85e-943e-4d8a-a2c4-d517bad23734\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") "
Nov 03 18:21:17.604987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.604827    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-tmp-dir\") pod \"829de85e-943e-4d8a-a2c4-d517bad23734\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") "
Nov 03 18:21:17.604987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.604908    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-dshm\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-dshm\") pod \"829de85e-943e-4d8a-a2c4-d517bad23734\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") "
Nov 03 18:21:17.604987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.604935    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829de85e-943e-4d8a-a2c4-d517bad23734-kube-api-access-5thf7\" (UniqueName: \"kubernetes.io/projected/829de85e-943e-4d8a-a2c4-d517bad23734-kube-api-access-5thf7\") pod \"829de85e-943e-4d8a-a2c4-d517bad23734\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") "
Nov 03 18:21:17.605196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.604989    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-home\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-home\") pod \"829de85e-943e-4d8a-a2c4-d517bad23734\" (UID: \"829de85e-943e-4d8a-a2c4-d517bad23734\") "
Nov 03 18:21:17.605196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.605107    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-model-cache" pod "829de85e-943e-4d8a-a2c4-d517bad23734" (UID: "829de85e-943e-4d8a-a2c4-d517bad23734"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:17.605196 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.605144    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-tmp-dir" pod "829de85e-943e-4d8a-a2c4-d517bad23734" (UID: "829de85e-943e-4d8a-a2c4-d517bad23734"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:17.605457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.605397    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-home" pod "829de85e-943e-4d8a-a2c4-d517bad23734" (UID: "829de85e-943e-4d8a-a2c4-d517bad23734"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:17.605457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.605420    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:17.605457 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.605441    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:17.607392 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.607363    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-dshm" pod "829de85e-943e-4d8a-a2c4-d517bad23734" (UID: "829de85e-943e-4d8a-a2c4-d517bad23734"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:17.607648 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.607623    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829de85e-943e-4d8a-a2c4-d517bad23734-tls-certs" pod "829de85e-943e-4d8a-a2c4-d517bad23734" (UID: "829de85e-943e-4d8a-a2c4-d517bad23734"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:21:17.607750 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.607641    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829de85e-943e-4d8a-a2c4-d517bad23734-kube-api-access-5thf7" pod "829de85e-943e-4d8a-a2c4-d517bad23734" (UID: "829de85e-943e-4d8a-a2c4-d517bad23734"). InnerVolumeSpecName "kube-api-access-5thf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:17.648423 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.648392    4493 generic.go:356] "Generic (PLEG): container finished" podID="829de85e-943e-4d8a-a2c4-d517bad23734" containerID="40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d" exitCode=0
Nov 03 18:21:17.648565 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.648434    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"
Nov 03 18:21:17.648565 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.648485    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp" event={"ID":"829de85e-943e-4d8a-a2c4-d517bad23734","Type":"ContainerDied","Data":"40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d"}
Nov 03 18:21:17.648565 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.648533    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp" event={"ID":"829de85e-943e-4d8a-a2c4-d517bad23734","Type":"ContainerDied","Data":"df4a9d1114c4c0978244eb47b22427f87c58132168a3101979579b495418941f"}
Nov 03 18:21:17.648565 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.648550    4493 scope.go:122] "RemoveContainer" containerID="40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d"
Nov 03 18:21:17.658955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.658937    4493 scope.go:122] "RemoveContainer" containerID="40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d"
Nov 03 18:21:17.659239 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:21:17.659221    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d\": container with ID starting with 40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d not found: ID does not exist" containerID="40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d"
Nov 03 18:21:17.659315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.659248    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d"} err="failed to get container status \"40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d\": rpc error: code = NotFound desc = could not find container \"40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d\": container with ID starting with 40db0ac6ce8094cbc1ff90ef9501b822c7dae610c439b313a571aea277afd08d not found: ID does not exist"
Nov 03 18:21:17.706330 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.706296    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:17.706330 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.706326    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thf7\" (UniqueName: \"kubernetes.io/projected/829de85e-943e-4d8a-a2c4-d517bad23734-kube-api-access-5thf7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:17.706330 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.706337    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/829de85e-943e-4d8a-a2c4-d517bad23734-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:17.706618 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.706345    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829de85e-943e-4d8a-a2c4-d517bad23734-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:17.779207 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.779172    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"]
Nov 03 18:21:17.784206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:17.784175    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0adf17/test-model-0adf17-kserve-6fd969dddf-t4prp"]
Nov 03 18:21:18.710808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:18.710771    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829de85e-943e-4d8a-a2c4-d517bad23734" path="/var/lib/kubelet/pods/829de85e-943e-4d8a-a2c4-d517bad23734/volumes"
Nov 03 18:21:26.608398 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:26.608361    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"]
Nov 03 18:21:26.660051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:26.660019    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-73a606-dffwd"]
Nov 03 18:21:26.714699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:26.714658    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b218d4b-55a7-40ad-8c98-bec70606d7d4" path="/var/lib/kubelet/pods/1b218d4b-55a7-40ad-8c98-bec70606d7d4/volumes"
Nov 03 18:21:33.318715 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:33.318677    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"]
Nov 03 18:21:33.369237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:33.369197    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ca6b33-cx2qf"]
Nov 03 18:21:34.711780 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:34.711747    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38b02b7-f02f-4407-a97b-5afcc760d0b6" path="/var/lib/kubelet/pods/e38b02b7-f02f-4407-a97b-5afcc760d0b6/volumes"
Nov 03 18:21:35.311727 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:35.311687    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"]
Nov 03 18:21:35.311987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:35.311950    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy" containerID="cri-o://2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230" gracePeriod=30
Nov 03 18:21:36.991695 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:21:36.991660    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:21:40.559155 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.559129    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 18:21:40.638046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.637974    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84ab85b-542a-40f5-9712-4516c3799a06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84ab85b-542a-40f5-9712-4516c3799a06-istiod-ca-cert\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638017    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-envoy\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638067    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-istio-token\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-istio-token\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638113    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e84ab85b-542a-40f5-9712-4516c3799a06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84ab85b-542a-40f5-9712-4516c3799a06-istio-podinfo\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638145    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-kube-api-access-wrn2t\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-kube-api-access-wrn2t\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638167    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-socket\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638204    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-credential-socket\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638230    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-certs\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638266    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-data\") pod \"e84ab85b-542a-40f5-9712-4516c3799a06\" (UID: \"e84ab85b-542a-40f5-9712-4516c3799a06\") "
Nov 03 18:21:40.638598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638339    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84ab85b-542a-40f5-9712-4516c3799a06-istiod-ca-cert" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:40.638598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638550    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-credential-socket" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:40.638772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638688    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.638772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638710    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84ab85b-542a-40f5-9712-4516c3799a06-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.638772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638758    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-data" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:40.639136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638768    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-certs" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:40.639136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.638811    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-socket" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:40.640535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.640504    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e84ab85b-542a-40f5-9712-4516c3799a06-istio-podinfo" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:21:40.640660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.640569    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-kube-api-access-wrn2t" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "kube-api-access-wrn2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:40.640891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.640856    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-envoy" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:21:40.640969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.640860    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-istio-token" pod "e84ab85b-542a-40f5-9712-4516c3799a06" (UID: "e84ab85b-542a-40f5-9712-4516c3799a06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:40.739763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.739733    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84ab85b-542a-40f5-9712-4516c3799a06-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.739763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.739762    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrn2t\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-kube-api-access-wrn2t\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.739983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.739777    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.739983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.739791    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.739983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.739804    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.739983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.739814    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84ab85b-542a-40f5-9712-4516c3799a06-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.739983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.739828    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84ab85b-542a-40f5-9712-4516c3799a06-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:40.783291 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.783258    4493 generic.go:356] "Generic (PLEG): container finished" podID="e84ab85b-542a-40f5-9712-4516c3799a06" containerID="2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230" exitCode=0
Nov 03 18:21:40.783470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.783302    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"
Nov 03 18:21:40.783470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.783342    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" event={"ID":"e84ab85b-542a-40f5-9712-4516c3799a06","Type":"ContainerDied","Data":"2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230"}
Nov 03 18:21:40.783470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.783397    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6" event={"ID":"e84ab85b-542a-40f5-9712-4516c3799a06","Type":"ContainerDied","Data":"54088df8d51b968c02581302e1b213b251d1e27e08ef0332a4ca3dfd472ffc8c"}
Nov 03 18:21:40.783470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.783419    4493 scope.go:122] "RemoveContainer" containerID="2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230"
Nov 03 18:21:40.792931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.792912    4493 scope.go:122] "RemoveContainer" containerID="2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230"
Nov 03 18:21:40.793201 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:21:40.793182    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230\": container with ID starting with 2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230 not found: ID does not exist" containerID="2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230"
Nov 03 18:21:40.793255 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.793211    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230"} err="failed to get container status \"2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230\": rpc error: code = NotFound desc = could not find container \"2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230\": container with ID starting with 2e295b8be90929657babb099caec34792c876a6508c299ef259ba9d136862230 not found: ID does not exist"
Nov 03 18:21:40.910564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.910484    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"]
Nov 03 18:21:40.919059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:40.919032    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-hswm6"]
Nov 03 18:21:42.711283 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:42.711247    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" path="/var/lib/kubelet/pods/e84ab85b-542a-40f5-9712-4516c3799a06/volumes"
Nov 03 18:21:43.104254 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104220    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"]
Nov 03 18:21:43.104623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104610    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e38b02b7-f02f-4407-a97b-5afcc760d0b6" containerName="revoke-keys"
Nov 03 18:21:43.104670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104625    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38b02b7-f02f-4407-a97b-5afcc760d0b6" containerName="revoke-keys"
Nov 03 18:21:43.104670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104644    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3b2fea08-041f-41d9-95be-811f63cc4434" containerName="payload-processing"
Nov 03 18:21:43.104670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104650    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2fea08-041f-41d9-95be-811f63cc4434" containerName="payload-processing"
Nov 03 18:21:43.104670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104659    4493 cpu_manager.go:414] "containerMap: removing container" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104686    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104699    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7786ff62-727a-4d61-98f6-e3f1071c97f7" containerName="payload-pre-processing"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104704    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7786ff62-727a-4d61-98f6-e3f1071c97f7" containerName="payload-pre-processing"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104713    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104717    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104727    4493 cpu_manager.go:414] "containerMap: removing container" podUID="829de85e-943e-4d8a-a2c4-d517bad23734" containerName="main"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104732    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829de85e-943e-4d8a-a2c4-d517bad23734" containerName="main"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104742    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7eb97c6b-8697-4a9b-af7a-f5d760e26543" containerName="revoke-keys"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104746    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb97c6b-8697-4a9b-af7a-f5d760e26543" containerName="revoke-keys"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104807    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38b02b7-f02f-4407-a97b-5afcc760d0b6" containerName="revoke-keys"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104819    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2fea08-041f-41d9-95be-811f63cc4434" containerName="payload-processing"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104827    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84ab85b-542a-40f5-9712-4516c3799a06" containerName="istio-proxy"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104837    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7786ff62-727a-4d61-98f6-e3f1071c97f7" containerName="payload-pre-processing"
Nov 03 18:21:43.104842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104847    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcac96ab-c8c5-4b4a-b5e6-6575bec12de2" containerName="istio-proxy"
Nov 03 18:21:43.105289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104853    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="829de85e-943e-4d8a-a2c4-d517bad23734" containerName="main"
Nov 03 18:21:43.105289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.104861    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb97c6b-8697-4a9b-af7a-f5d760e26543" containerName="revoke-keys"
Nov 03 18:21:43.109422 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.109392    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.164835 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.164805    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gcf\" (UniqueName: \"kubernetes.io/projected/fe741fa4-faaf-4da2-b97d-9927063ba929-kube-api-access-d8gcf\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.165005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.164943    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.165005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.164985    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.165521 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.165506    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:21:43.171060 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.171036    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"]
Nov 03 18:21:43.200947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.200921    4493 scope.go:122] "RemoveContainer" containerID="3c858a814a99383f959dcd4c1f28f2b9c9b3295413f8c3c267e1daadd1950151"
Nov 03 18:21:43.210969 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.210941    4493 scope.go:122] "RemoveContainer" containerID="3be6609e031fe702e481570e9172b9e0d5f75c759ca996e7313ba009c34e96b8"
Nov 03 18:21:43.216350 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.216331    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:21:43.216444 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.216377    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:21:43.217710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.217694    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:21:43.265586 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.265560    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gcf\" (UniqueName: \"kubernetes.io/projected/fe741fa4-faaf-4da2-b97d-9927063ba929-kube-api-access-d8gcf\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.265692 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.265654    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.265751 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.265691    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.266357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.266333    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.268580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.268554    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.328849 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.328820    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gcf\" (UniqueName: \"kubernetes.io/projected/fe741fa4-faaf-4da2-b97d-9927063ba929-kube-api-access-d8gcf\") pod \"maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.421853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.421787    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:43.799103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:43.799075    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"]
Nov 03 18:21:43.800819 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:21:43.800784    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe741fa4_faaf_4da2_b97d_9927063ba929.slice/crio-2c5f8af55ad6719523850ac80c8f0a85c8daa4f2b8e346e208c2ec0533c6ec6e WatchSource:0}: Error finding container 2c5f8af55ad6719523850ac80c8f0a85c8daa4f2b8e346e208c2ec0533c6ec6e: Status 404 returned error can't find the container with id 2c5f8af55ad6719523850ac80c8f0a85c8daa4f2b8e346e208c2ec0533c6ec6e
Nov 03 18:21:44.808647 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:44.808603    4493 generic.go:356] "Generic (PLEG): container finished" podID="fe741fa4-faaf-4da2-b97d-9927063ba929" containerID="e9c5c36e90a15253bf837d8eb62c07ea71340609372ec7c15f5e9476b8855890" exitCode=0
Nov 03 18:21:44.809083 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:44.808700    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg" event={"ID":"fe741fa4-faaf-4da2-b97d-9927063ba929","Type":"ContainerDied","Data":"e9c5c36e90a15253bf837d8eb62c07ea71340609372ec7c15f5e9476b8855890"}
Nov 03 18:21:44.809083 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:44.808759    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg" event={"ID":"fe741fa4-faaf-4da2-b97d-9927063ba929","Type":"ContainerStarted","Data":"2c5f8af55ad6719523850ac80c8f0a85c8daa4f2b8e346e208c2ec0533c6ec6e"}
Nov 03 18:21:45.949822 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:45.949798    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:45.991425 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:45.991398    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-openshift-ca\") pod \"fe741fa4-faaf-4da2-b97d-9927063ba929\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") "
Nov 03 18:21:45.991598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:45.991439    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-xks-ca\") pod \"fe741fa4-faaf-4da2-b97d-9927063ba929\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") "
Nov 03 18:21:45.991598 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:45.991477    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe741fa4-faaf-4da2-b97d-9927063ba929-kube-api-access-d8gcf\" (UniqueName: \"kubernetes.io/projected/fe741fa4-faaf-4da2-b97d-9927063ba929-kube-api-access-d8gcf\") pod \"fe741fa4-faaf-4da2-b97d-9927063ba929\" (UID: \"fe741fa4-faaf-4da2-b97d-9927063ba929\") "
Nov 03 18:21:45.991762 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:45.991738    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-openshift-ca" pod "fe741fa4-faaf-4da2-b97d-9927063ba929" (UID: "fe741fa4-faaf-4da2-b97d-9927063ba929"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:21:45.994260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:45.994230    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-xks-ca" pod "fe741fa4-faaf-4da2-b97d-9927063ba929" (UID: "fe741fa4-faaf-4da2-b97d-9927063ba929"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:21:45.994396 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:45.994373    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe741fa4-faaf-4da2-b97d-9927063ba929-kube-api-access-d8gcf" pod "fe741fa4-faaf-4da2-b97d-9927063ba929" (UID: "fe741fa4-faaf-4da2-b97d-9927063ba929"). InnerVolumeSpecName "kube-api-access-d8gcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:21:46.092520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:46.092439    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:46.092520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:46.092472    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe741fa4-faaf-4da2-b97d-9927063ba929-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:46.092520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:46.092482    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8gcf\" (UniqueName: \"kubernetes.io/projected/fe741fa4-faaf-4da2-b97d-9927063ba929-kube-api-access-d8gcf\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:21:46.819469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:46.819434    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"
Nov 03 18:21:46.819469 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:46.819456    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg" event={"ID":"fe741fa4-faaf-4da2-b97d-9927063ba929","Type":"ContainerDied","Data":"2c5f8af55ad6719523850ac80c8f0a85c8daa4f2b8e346e208c2ec0533c6ec6e"}
Nov 03 18:21:46.819691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:21:46.819513    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5f8af55ad6719523850ac80c8f0a85c8daa4f2b8e346e208c2ec0533c6ec6e"
Nov 03 18:22:05.823283 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:05.823243    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"]
Nov 03 18:22:05.835623 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:05.835599    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adf17-lh6hg"]
Nov 03 18:22:06.711103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:06.711065    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe741fa4-faaf-4da2-b97d-9927063ba929" path="/var/lib/kubelet/pods/fe741fa4-faaf-4da2-b97d-9927063ba929/volumes"
Nov 03 18:22:06.991322 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:06.991281    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:22:14.880898 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.880840    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"]
Nov 03 18:22:14.881306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.881230    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fe741fa4-faaf-4da2-b97d-9927063ba929" containerName="revoke-keys"
Nov 03 18:22:14.881306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.881242    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe741fa4-faaf-4da2-b97d-9927063ba929" containerName="revoke-keys"
Nov 03 18:22:14.881306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.881305    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe741fa4-faaf-4da2-b97d-9927063ba929" containerName="revoke-keys"
Nov 03 18:22:14.884322 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.884305    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:14.979533 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.979494    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:14.979706 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.979556    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:14.979757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.979714    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68xd\" (UniqueName: \"kubernetes.io/projected/43bd9871-a606-4da8-9f68-6daa52a25882-kube-api-access-q68xd\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:14.990288 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.990259    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:22:14.990439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.990422    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:22:14.990520 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.990479    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:22:14.991114 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.991097    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:22:14.992475 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:14.992456    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"]
Nov 03 18:22:15.080750 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.080716    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68xd\" (UniqueName: \"kubernetes.io/projected/43bd9871-a606-4da8-9f68-6daa52a25882-kube-api-access-q68xd\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:15.080954 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.080795    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:15.080954 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.080829    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:15.081481 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.081463    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:15.083311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.083295    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:15.144406 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.144335    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68xd\" (UniqueName: \"kubernetes.io/projected/43bd9871-a606-4da8-9f68-6daa52a25882-kube-api-access-q68xd\") pod \"maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:15.196522 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.196488    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:15.571717 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.571692    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"]
Nov 03 18:22:15.982001 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.981966    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z" event={"ID":"43bd9871-a606-4da8-9f68-6daa52a25882","Type":"ContainerStarted","Data":"4a9edd04cea2ab24f3589fe4442d71b8533580e743a41f3a82870e904fb9da70"}
Nov 03 18:22:15.982438 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:15.982017    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z" event={"ID":"43bd9871-a606-4da8-9f68-6daa52a25882","Type":"ContainerStarted","Data":"a8db583527af7c3e529155cad8ba0a235558c8cacca3a81bd1144c446b9685a8"}
Nov 03 18:22:16.102114 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:16.102027    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z" podStartSLOduration=2.102014774 podStartE2EDuration="2.102014774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:22:16.101495656 +0000 UTC m=+2915.560666643" watchObservedRunningTime="2026-11-03 18:22:16.102014774 +0000 UTC m=+2915.561185759"
Nov 03 18:22:16.988359 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:16.988323    4493 generic.go:356] "Generic (PLEG): container finished" podID="43bd9871-a606-4da8-9f68-6daa52a25882" containerID="4a9edd04cea2ab24f3589fe4442d71b8533580e743a41f3a82870e904fb9da70" exitCode=0
Nov 03 18:22:16.988743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:16.988413    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z" event={"ID":"43bd9871-a606-4da8-9f68-6daa52a25882","Type":"ContainerDied","Data":"4a9edd04cea2ab24f3589fe4442d71b8533580e743a41f3a82870e904fb9da70"}
Nov 03 18:22:17.090373 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:17.090339    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"]
Nov 03 18:22:17.094980 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:17.094959    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9b5fad-vdmjx"]
Nov 03 18:22:18.064451 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.064418    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"]
Nov 03 18:22:18.113721 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.113691    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d079df1-rqb85"]
Nov 03 18:22:18.133062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.133042    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:18.210564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.210533    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bd9871-a606-4da8-9f68-6daa52a25882-kube-api-access-q68xd\" (UniqueName: \"kubernetes.io/projected/43bd9871-a606-4da8-9f68-6daa52a25882-kube-api-access-q68xd\") pod \"43bd9871-a606-4da8-9f68-6daa52a25882\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") "
Nov 03 18:22:18.210719 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.210587    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-openshift-ca\") pod \"43bd9871-a606-4da8-9f68-6daa52a25882\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") "
Nov 03 18:22:18.210719 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.210615    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-xks-ca\") pod \"43bd9871-a606-4da8-9f68-6daa52a25882\" (UID: \"43bd9871-a606-4da8-9f68-6daa52a25882\") "
Nov 03 18:22:18.210987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.210964    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-openshift-ca" pod "43bd9871-a606-4da8-9f68-6daa52a25882" (UID: "43bd9871-a606-4da8-9f68-6daa52a25882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:18.212777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.212751    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-xks-ca" pod "43bd9871-a606-4da8-9f68-6daa52a25882" (UID: "43bd9871-a606-4da8-9f68-6daa52a25882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:18.212777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.212772    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bd9871-a606-4da8-9f68-6daa52a25882-kube-api-access-q68xd" pod "43bd9871-a606-4da8-9f68-6daa52a25882" (UID: "43bd9871-a606-4da8-9f68-6daa52a25882"). InnerVolumeSpecName "kube-api-access-q68xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:18.311430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.311358    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q68xd\" (UniqueName: \"kubernetes.io/projected/43bd9871-a606-4da8-9f68-6daa52a25882-kube-api-access-q68xd\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:18.311430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.311389    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:18.311430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.311399    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd9871-a606-4da8-9f68-6daa52a25882-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:18.711533 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.711454    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a089dcc-953c-4ef4-9889-b945702e8bf8" path="/var/lib/kubelet/pods/9a089dcc-953c-4ef4-9889-b945702e8bf8/volumes"
Nov 03 18:22:18.711862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:18.711848    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52ebf0a-23bd-45de-a224-9feed6254685" path="/var/lib/kubelet/pods/d52ebf0a-23bd-45de-a224-9feed6254685/volumes"
Nov 03 18:22:19.005791 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.005761    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"
Nov 03 18:22:19.005982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.005794    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z" event={"ID":"43bd9871-a606-4da8-9f68-6daa52a25882","Type":"ContainerDied","Data":"a8db583527af7c3e529155cad8ba0a235558c8cacca3a81bd1144c446b9685a8"}
Nov 03 18:22:19.005982 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.005843    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8db583527af7c3e529155cad8ba0a235558c8cacca3a81bd1144c446b9685a8"
Nov 03 18:22:19.271543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.271452    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"]
Nov 03 18:22:19.277718 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.277692    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"]
Nov 03 18:22:19.321658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.321625    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"]
Nov 03 18:22:19.706785 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.706682    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"]
Nov 03 18:22:19.707006 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.706952    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy" containerID="cri-o://000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b" gracePeriod=30
Nov 03 18:22:19.708608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.708583    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"]
Nov 03 18:22:19.708787 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.708766    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy" containerID="cri-o://02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1" gracePeriod=30
Nov 03 18:22:19.710288 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.710265    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"]
Nov 03 18:22:19.710506 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.710476    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" containerID="cri-o://c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609" gracePeriod=30
Nov 03 18:22:19.711955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.711934    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"]
Nov 03 18:22:19.712122 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:19.712104    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy" containerID="cri-o://71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4" gracePeriod=30
Nov 03 18:22:22.946639 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:22.946547    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-hjbns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 18:22:22.946639 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:22.946608    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 18:22:24.989678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:24.989648    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 18:22:25.033107 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.033082    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 18:22:25.037131 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.037107    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:22:25.039664 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.039636    4493 generic.go:356] "Generic (PLEG): container finished" podID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerID="c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609" exitCode=0
Nov 03 18:22:25.039771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.039666    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" event={"ID":"9bcac327-d2fe-4d33-bc16-ebd373d687f3","Type":"ContainerDied","Data":"c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609"}
Nov 03 18:22:25.039771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.039687    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"
Nov 03 18:22:25.039771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.039732    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns" event={"ID":"9bcac327-d2fe-4d33-bc16-ebd373d687f3","Type":"ContainerDied","Data":"6ee4cc83bd55cd3453025f3f456c6e87230432e8b7a54d2542fde4e56fa501b5"}
Nov 03 18:22:25.039771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.039763    4493 scope.go:122] "RemoveContainer" containerID="c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609"
Nov 03 18:22:25.040691 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.040670    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:22:25.040961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.040942    4493 generic.go:356] "Generic (PLEG): container finished" podID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerID="000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b" exitCode=0
Nov 03 18:22:25.041036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.040988    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"
Nov 03 18:22:25.041036 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.041001    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" event={"ID":"7e0b680c-1d27-42b4-b971-8377d277d16f","Type":"ContainerDied","Data":"000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b"}
Nov 03 18:22:25.041145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.041047    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j" event={"ID":"7e0b680c-1d27-42b4-b971-8377d277d16f","Type":"ContainerDied","Data":"287f4668491e29cff8897d9ecf55804d9d94fdcb666f7149ad55ebeb69b210e8"}
Nov 03 18:22:25.042476 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.042452    4493 generic.go:356] "Generic (PLEG): container finished" podID="929153f0-5444-407c-a543-7650ecebdc93" containerID="71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4" exitCode=0
Nov 03 18:22:25.042587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.042487    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"
Nov 03 18:22:25.042587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.042529    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" event={"ID":"929153f0-5444-407c-a543-7650ecebdc93","Type":"ContainerDied","Data":"71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4"}
Nov 03 18:22:25.042587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.042574    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n" event={"ID":"929153f0-5444-407c-a543-7650ecebdc93","Type":"ContainerDied","Data":"04d2d6c59078e1d4a8c487326be99625518a46697838cb2bd48e129ce700f66b"}
Nov 03 18:22:25.043947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.043926    4493 generic.go:356] "Generic (PLEG): container finished" podID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerID="02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1" exitCode=0
Nov 03 18:22:25.044042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.043958    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" event={"ID":"ecf715a9-6cd9-4a37-844b-ee1356f752f2","Type":"ContainerDied","Data":"02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1"}
Nov 03 18:22:25.044042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.043984    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d" event={"ID":"ecf715a9-6cd9-4a37-844b-ee1356f752f2","Type":"ContainerDied","Data":"cbdec06a071d4fd239918edf1b3f316d0a72eb4af038c9257234c55fdf2fa5f7"}
Nov 03 18:22:25.044042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.043960    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"
Nov 03 18:22:25.049332 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.049315    4493 scope.go:122] "RemoveContainer" containerID="c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609"
Nov 03 18:22:25.049639 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:25.049618    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609\": container with ID starting with c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609 not found: ID does not exist" containerID="c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609"
Nov 03 18:22:25.049699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.049651    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609"} err="failed to get container status \"c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609\": rpc error: code = NotFound desc = could not find container \"c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609\": container with ID starting with c62a365a8ca941d6880e0f8941f6282d936235bdf7e1a37cf5c16057cdb92609 not found: ID does not exist"
Nov 03 18:22:25.049699 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.049670    4493 scope.go:122] "RemoveContainer" containerID="000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b"
Nov 03 18:22:25.058135 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.058113    4493 scope.go:122] "RemoveContainer" containerID="000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b"
Nov 03 18:22:25.058391 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:25.058372    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b\": container with ID starting with 000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b not found: ID does not exist" containerID="000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b"
Nov 03 18:22:25.058483 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.058399    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b"} err="failed to get container status \"000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b\": rpc error: code = NotFound desc = could not find container \"000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b\": container with ID starting with 000126f8e3ad0c50bb99f4fe3c520bcb5a828ffd3c6891c271f109a7cf52e01b not found: ID does not exist"
Nov 03 18:22:25.058483 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.058416    4493 scope.go:122] "RemoveContainer" containerID="71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4"
Nov 03 18:22:25.066628 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.066606    4493 scope.go:122] "RemoveContainer" containerID="71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4"
Nov 03 18:22:25.066909 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:25.066859    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4\": container with ID starting with 71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4 not found: ID does not exist" containerID="71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4"
Nov 03 18:22:25.067021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.066918    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4"} err="failed to get container status \"71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4\": rpc error: code = NotFound desc = could not find container \"71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4\": container with ID starting with 71d58cc9e28f73ddcb03f2264081d022d087caec04aebfcab51e3c8b59d169f4 not found: ID does not exist"
Nov 03 18:22:25.067021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.066939    4493 scope.go:122] "RemoveContainer" containerID="02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1"
Nov 03 18:22:25.074861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.074844    4493 scope.go:122] "RemoveContainer" containerID="02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1"
Nov 03 18:22:25.075112 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:25.075094    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1\": container with ID starting with 02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1 not found: ID does not exist" containerID="02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1"
Nov 03 18:22:25.075177 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.075123    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1"} err="failed to get container status \"02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1\": rpc error: code = NotFound desc = could not find container \"02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1\": container with ID starting with 02612ed0f8dbc759023ab7f50b6e857c3c20dc162563a3b78db80648f9797fa1 not found: ID does not exist"
Nov 03 18:22:25.079434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079415    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/929153f0-5444-407c-a543-7650ecebdc93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929153f0-5444-407c-a543-7650ecebdc93-istio-podinfo\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.079494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079459    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-envoy\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.079494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079477    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0b680c-1d27-42b4-b971-8377d277d16f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b680c-1d27-42b4-b971-8377d277d16f-istiod-ca-cert\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.079568 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079499    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-envoy\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.079568 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079534    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-data\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.079669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079572    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-envoy\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.079669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079604    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-certs\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.079669 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079643    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-token\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-token\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.079842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079672    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-credential-socket\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.079842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079699    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-kube-api-access-ztzl6\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-kube-api-access-ztzl6\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.079842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079725    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-socket\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.079842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079750    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-podinfo\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.079842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079779    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-socket\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.079842 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079812    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-data" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.080200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079818    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-credential-socket\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.080200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079855    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0b680c-1d27-42b4-b971-8377d277d16f-istiod-ca-cert" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:25.080200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.079981    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-credential-socket\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.080200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080023    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-podinfo\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.080200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080028    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-credential-socket" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.080200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080054    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929153f0-5444-407c-a543-7650ecebdc93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929153f0-5444-407c-a543-7650ecebdc93-istiod-ca-cert\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.080200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080141    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-podinfo\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080213    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-credential-socket" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080219    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-certs" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080199    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-certs\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080306    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-data\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080333    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-certs\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080380    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-data\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080455    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-kube-api-access-pvw8l\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-kube-api-access-pvw8l\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080489    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-data\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080524    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-envoy\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080560    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istiod-ca-cert\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080600    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-certs\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080641    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-kube-api-access-j782z\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-kube-api-access-j782z\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080668    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-socket\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080714    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-kube-api-access-6w2hg\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-kube-api-access-6w2hg\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.080784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080746    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-token\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-token\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080798    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-socket\") pod \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\" (UID: \"ecf715a9-6cd9-4a37-844b-ee1356f752f2\") "
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080833    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-istio-token\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-istio-token\") pod \"929153f0-5444-407c-a543-7650ecebdc93\" (UID: \"929153f0-5444-407c-a543-7650ecebdc93\") "
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080891    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istiod-ca-cert\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080917    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-token\") pod \"7e0b680c-1d27-42b4-b971-8377d277d16f\" (UID: \"7e0b680c-1d27-42b4-b971-8377d277d16f\") "
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.080980    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-credential-socket\") pod \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\" (UID: \"9bcac327-d2fe-4d33-bc16-ebd373d687f3\") "
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081015    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-certs" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081221    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-credential-socket" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081225    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-socket" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081454    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-credential-socket" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081462    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081486    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081502    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081516    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081518    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-data" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081530    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081545    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081559    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b680c-1d27-42b4-b971-8377d277d16f-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081573    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.081640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081648    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-socket" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.082770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081753    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-data" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.082770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081773    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-certs" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.082770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.081993    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-data" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.082770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.082032    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-socket" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.084044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.083494    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-envoy" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.084044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.083505    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/929153f0-5444-407c-a543-7650ecebdc93-istio-podinfo" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:22:25.084044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.083711    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-socket" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.084044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.083758    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-certs" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.084044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.083770    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-envoy" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.084044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.083890    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929153f0-5444-407c-a543-7650ecebdc93-istiod-ca-cert" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:25.084439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.084086    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istiod-ca-cert" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:25.084439 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.084241    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istiod-ca-cert" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:25.085091 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.085037    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-kube-api-access-ztzl6" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "kube-api-access-ztzl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.085470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.085441    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-podinfo" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:22:25.085987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.085680    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-podinfo" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:22:25.085987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.085687    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-kube-api-access-6w2hg" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "kube-api-access-6w2hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.085987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.085757    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-token" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.085987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.085865    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-podinfo" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:22:25.085987 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.085948    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-kube-api-access-j782z" pod "ecf715a9-6cd9-4a37-844b-ee1356f752f2" (UID: "ecf715a9-6cd9-4a37-844b-ee1356f752f2"). InnerVolumeSpecName "kube-api-access-j782z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.086345 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.086167    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-envoy" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.086345 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.086258    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-envoy" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:25.086609 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.086583    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-token" pod "9bcac327-d2fe-4d33-bc16-ebd373d687f3" (UID: "9bcac327-d2fe-4d33-bc16-ebd373d687f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.086977 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.086913    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-token" pod "7e0b680c-1d27-42b4-b971-8377d277d16f" (UID: "7e0b680c-1d27-42b4-b971-8377d277d16f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.087149 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.087131    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-kube-api-access-pvw8l" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "kube-api-access-pvw8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.087333 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.087317    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-istio-token" pod "929153f0-5444-407c-a543-7650ecebdc93" (UID: "929153f0-5444-407c-a543-7650ecebdc93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:25.182738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182705    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvw8l\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-kube-api-access-pvw8l\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182738 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182733    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182759    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182770    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182781    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182791    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j782z\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-kube-api-access-j782z\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182800    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182808    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2hg\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-kube-api-access-6w2hg\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182816    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182825    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182833    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929153f0-5444-407c-a543-7650ecebdc93-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182840    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182849    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182857    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcac327-d2fe-4d33-bc16-ebd373d687f3-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182865    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/929153f0-5444-407c-a543-7650ecebdc93-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182902    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182911    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182919    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182927    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182935    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztzl6\" (UniqueName: \"kubernetes.io/projected/7e0b680c-1d27-42b4-b971-8377d277d16f-kube-api-access-ztzl6\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182943    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcac327-d2fe-4d33-bc16-ebd373d687f3-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182951    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b680c-1d27-42b4-b971-8377d277d16f-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182958    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182966    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/929153f0-5444-407c-a543-7650ecebdc93-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.182966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182974    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b680c-1d27-42b4-b971-8377d277d16f-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.183651 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182982    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.183651 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.182991    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf715a9-6cd9-4a37-844b-ee1356f752f2-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.183651 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.183000    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/929153f0-5444-407c-a543-7650ecebdc93-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:25.714334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.714311    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"]
Nov 03 18:22:25.720348 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.720322    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rww5d"]
Nov 03 18:22:25.807471 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807434    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"]
Nov 03 18:22:25.807771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807759    4493 cpu_manager.go:414] "containerMap: removing container" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy"
Nov 03 18:22:25.807820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807773    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy"
Nov 03 18:22:25.807820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807786    4493 cpu_manager.go:414] "containerMap: removing container" podUID="43bd9871-a606-4da8-9f68-6daa52a25882" containerName="revoke-keys"
Nov 03 18:22:25.807820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807791    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bd9871-a606-4da8-9f68-6daa52a25882" containerName="revoke-keys"
Nov 03 18:22:25.807820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807800    4493 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy"
Nov 03 18:22:25.807820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807808    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy"
Nov 03 18:22:25.807820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807816    4493 cpu_manager.go:414] "containerMap: removing container" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy"
Nov 03 18:22:25.807820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807821    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy"
Nov 03 18:22:25.808076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807829    4493 cpu_manager.go:414] "containerMap: removing container" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy"
Nov 03 18:22:25.808076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807834    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy"
Nov 03 18:22:25.808076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807916    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" containerName="istio-proxy"
Nov 03 18:22:25.808076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807924    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bd9871-a606-4da8-9f68-6daa52a25882" containerName="revoke-keys"
Nov 03 18:22:25.808076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807930    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" containerName="istio-proxy"
Nov 03 18:22:25.808076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807937    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" containerName="istio-proxy"
Nov 03 18:22:25.808076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.807942    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="929153f0-5444-407c-a543-7650ecebdc93" containerName="istio-proxy"
Nov 03 18:22:25.812764 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.812742    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.834703 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.834679    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"]
Nov 03 18:22:25.840486 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.840461    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-hjbns"]
Nov 03 18:22:25.869048 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.869021    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:22:25.869186 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.869087    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:22:25.870564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.870542    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"]
Nov 03 18:22:25.883229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.883207    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:22:25.883355 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.883286    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:22:25.889613 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.889586    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.889689 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.889669    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.889770 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.889751    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sptbk\" (UniqueName: \"kubernetes.io/projected/288a7e72-ae62-46e0-80c6-b5d27c26edb1-kube-api-access-sptbk\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.954722 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.954693    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"]
Nov 03 18:22:25.960237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.960210    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q2k5n"]
Nov 03 18:22:25.990793 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.990726    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.990793 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.990780    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sptbk\" (UniqueName: \"kubernetes.io/projected/288a7e72-ae62-46e0-80c6-b5d27c26edb1-kube-api-access-sptbk\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.991187 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.990840    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.991455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.991437    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:25.993208 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:25.993188    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:26.054772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.054743    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sptbk\" (UniqueName: \"kubernetes.io/projected/288a7e72-ae62-46e0-80c6-b5d27c26edb1-kube-api-access-sptbk\") pod \"maas-api-revoke-sub-7373309d-kwfxp\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:26.073839 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.073809    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"]
Nov 03 18:22:26.079990 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.079964    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tlz9j"]
Nov 03 18:22:26.125494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.125466    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:26.385604 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.385529    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"]
Nov 03 18:22:26.494226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.494197    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"]
Nov 03 18:22:26.506269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.506231    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"]
Nov 03 18:22:26.508603 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:22:26.508576    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a7e72_ae62_46e0_80c6_b5d27c26edb1.slice/crio-448b6e99c0cd240865982fc5e3d9094e34ca5d551946bb656239d5c2797e8c91 WatchSource:0}: Error finding container 448b6e99c0cd240865982fc5e3d9094e34ca5d551946bb656239d5c2797e8c91: Status 404 returned error can't find the container with id 448b6e99c0cd240865982fc5e3d9094e34ca5d551946bb656239d5c2797e8c91
Nov 03 18:22:26.712362 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.712276    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0b680c-1d27-42b4-b971-8377d277d16f" path="/var/lib/kubelet/pods/7e0b680c-1d27-42b4-b971-8377d277d16f/volumes"
Nov 03 18:22:26.712861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.712842    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929153f0-5444-407c-a543-7650ecebdc93" path="/var/lib/kubelet/pods/929153f0-5444-407c-a543-7650ecebdc93/volumes"
Nov 03 18:22:26.713274 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.713256    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcac327-d2fe-4d33-bc16-ebd373d687f3" path="/var/lib/kubelet/pods/9bcac327-d2fe-4d33-bc16-ebd373d687f3/volumes"
Nov 03 18:22:26.714056 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:26.714041    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf715a9-6cd9-4a37-844b-ee1356f752f2" path="/var/lib/kubelet/pods/ecf715a9-6cd9-4a37-844b-ee1356f752f2/volumes"
Nov 03 18:22:27.063053 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:27.063016    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp" event={"ID":"288a7e72-ae62-46e0-80c6-b5d27c26edb1","Type":"ContainerStarted","Data":"c54324338f041020825ff39a2714b79f0c1fba37d00c28d2b06a4e040c9c7349"}
Nov 03 18:22:27.063412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:27.063079    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp" event={"ID":"288a7e72-ae62-46e0-80c6-b5d27c26edb1","Type":"ContainerStarted","Data":"448b6e99c0cd240865982fc5e3d9094e34ca5d551946bb656239d5c2797e8c91"}
Nov 03 18:22:27.184454 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:27.184402    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp" podStartSLOduration=2.184388807 podStartE2EDuration="2.184388807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:22:27.18351657 +0000 UTC m=+2926.642687556" watchObservedRunningTime="2026-11-03 18:22:27.184388807 +0000 UTC m=+2926.643559792"
Nov 03 18:22:28.070222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:28.070187    4493 generic.go:356] "Generic (PLEG): container finished" podID="288a7e72-ae62-46e0-80c6-b5d27c26edb1" containerID="c54324338f041020825ff39a2714b79f0c1fba37d00c28d2b06a4e040c9c7349" exitCode=0
Nov 03 18:22:28.070663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:28.070268    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp" event={"ID":"288a7e72-ae62-46e0-80c6-b5d27c26edb1","Type":"ContainerDied","Data":"c54324338f041020825ff39a2714b79f0c1fba37d00c28d2b06a4e040c9c7349"}
Nov 03 18:22:29.204972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.204945    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:29.322315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.322287    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-openshift-ca\") pod \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") "
Nov 03 18:22:29.322474 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.322322    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288a7e72-ae62-46e0-80c6-b5d27c26edb1-kube-api-access-sptbk\" (UniqueName: \"kubernetes.io/projected/288a7e72-ae62-46e0-80c6-b5d27c26edb1-kube-api-access-sptbk\") pod \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") "
Nov 03 18:22:29.322474 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.322389    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-xks-ca\") pod \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\" (UID: \"288a7e72-ae62-46e0-80c6-b5d27c26edb1\") "
Nov 03 18:22:29.322615 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.322589    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-openshift-ca" pod "288a7e72-ae62-46e0-80c6-b5d27c26edb1" (UID: "288a7e72-ae62-46e0-80c6-b5d27c26edb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:29.324972 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.324941    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288a7e72-ae62-46e0-80c6-b5d27c26edb1-kube-api-access-sptbk" pod "288a7e72-ae62-46e0-80c6-b5d27c26edb1" (UID: "288a7e72-ae62-46e0-80c6-b5d27c26edb1"). InnerVolumeSpecName "kube-api-access-sptbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:29.325358 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.325336    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-xks-ca" pod "288a7e72-ae62-46e0-80c6-b5d27c26edb1" (UID: "288a7e72-ae62-46e0-80c6-b5d27c26edb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:29.423237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.423202    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:29.423237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.423229    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sptbk\" (UniqueName: \"kubernetes.io/projected/288a7e72-ae62-46e0-80c6-b5d27c26edb1-kube-api-access-sptbk\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:29.423237 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:29.423239    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288a7e72-ae62-46e0-80c6-b5d27c26edb1-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:30.081843 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:30.081803    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"
Nov 03 18:22:30.081843 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:30.081824    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp" event={"ID":"288a7e72-ae62-46e0-80c6-b5d27c26edb1","Type":"ContainerDied","Data":"448b6e99c0cd240865982fc5e3d9094e34ca5d551946bb656239d5c2797e8c91"}
Nov 03 18:22:30.082128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:30.081896    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="448b6e99c0cd240865982fc5e3d9094e34ca5d551946bb656239d5c2797e8c91"
Nov 03 18:22:31.386485 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:31.386430    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" podUID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" containerName="payload-processing" containerID="cri-o://2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c" gracePeriod=25
Nov 03 18:22:31.494837 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:31.494802    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" podUID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" containerName="payload-pre-processing" containerID="cri-o://8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83" gracePeriod=25
Nov 03 18:22:31.794725 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:31.794691    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" podUID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:31.794896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:31.794687    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" podUID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:34.292096 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.291980    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main" containerID="cri-o://2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67" gracePeriod=45
Nov 03 18:22:34.299642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.299605    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main" containerID="cri-o://92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733" gracePeriod=45
Nov 03 18:22:34.340710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.340661    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main" containerID="cri-o://f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2" gracePeriod=45
Nov 03 18:22:34.592460 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.592101    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 18:22:34.621037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.621009    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 18:22:34.652059 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.652037    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 18:22:34.674961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.674939    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-dshm\") pod \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") "
Nov 03 18:22:34.675090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.674990    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-home\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-home\") pod \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") "
Nov 03 18:22:34.675090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675010    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-model-cache\") pod \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") "
Nov 03 18:22:34.675090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675039    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-home\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-home\") pod \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") "
Nov 03 18:22:34.675090 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675057    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-dshm\") pod \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") "
Nov 03 18:22:34.675321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675091    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tls-certs\") pod \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") "
Nov 03 18:22:34.675321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675119    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383193f6-170f-4f03-a9f8-4fd37e2e11e1-kube-api-access-b2x9r\" (UniqueName: \"kubernetes.io/projected/383193f6-170f-4f03-a9f8-4fd37e2e11e1-kube-api-access-b2x9r\") pod \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") "
Nov 03 18:22:34.675321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675176    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tls-certs\") pod \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") "
Nov 03 18:22:34.675321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675229    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-home" pod "be9f9fb3-acd8-486c-a33c-51c0c8a8f225" (UID: "be9f9fb3-acd8-486c-a33c-51c0c8a8f225"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.675321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675236    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-model-cache\") pod \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") "
Nov 03 18:22:34.675321 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675315    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-kube-api-access-j2j9h\" (UniqueName: \"kubernetes.io/projected/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-kube-api-access-j2j9h\") pod \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") "
Nov 03 18:22:34.675700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675361    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tmp-dir\") pod \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\" (UID: \"be9f9fb3-acd8-486c-a33c-51c0c8a8f225\") "
Nov 03 18:22:34.675700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675434    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tmp-dir\") pod \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\" (UID: \"383193f6-170f-4f03-a9f8-4fd37e2e11e1\") "
Nov 03 18:22:34.675955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675359    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-model-cache" pod "383193f6-170f-4f03-a9f8-4fd37e2e11e1" (UID: "383193f6-170f-4f03-a9f8-4fd37e2e11e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.675955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675414    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-model-cache" pod "be9f9fb3-acd8-486c-a33c-51c0c8a8f225" (UID: "be9f9fb3-acd8-486c-a33c-51c0c8a8f225"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.675955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675741    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-home" pod "383193f6-170f-4f03-a9f8-4fd37e2e11e1" (UID: "383193f6-170f-4f03-a9f8-4fd37e2e11e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.675955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675773    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.675955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675838    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tmp-dir" pod "be9f9fb3-acd8-486c-a33c-51c0c8a8f225" (UID: "be9f9fb3-acd8-486c-a33c-51c0c8a8f225"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.676293 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.675967    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tmp-dir" pod "383193f6-170f-4f03-a9f8-4fd37e2e11e1" (UID: "383193f6-170f-4f03-a9f8-4fd37e2e11e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.677453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.677424    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-dshm" pod "be9f9fb3-acd8-486c-a33c-51c0c8a8f225" (UID: "be9f9fb3-acd8-486c-a33c-51c0c8a8f225"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.677768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.677752    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-dshm" pod "383193f6-170f-4f03-a9f8-4fd37e2e11e1" (UID: "383193f6-170f-4f03-a9f8-4fd37e2e11e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.678069 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.678048    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tls-certs" pod "383193f6-170f-4f03-a9f8-4fd37e2e11e1" (UID: "383193f6-170f-4f03-a9f8-4fd37e2e11e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:34.678134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.678066    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tls-certs" pod "be9f9fb3-acd8-486c-a33c-51c0c8a8f225" (UID: "be9f9fb3-acd8-486c-a33c-51c0c8a8f225"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:34.678134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.678087    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383193f6-170f-4f03-a9f8-4fd37e2e11e1-kube-api-access-b2x9r" pod "383193f6-170f-4f03-a9f8-4fd37e2e11e1" (UID: "383193f6-170f-4f03-a9f8-4fd37e2e11e1"). InnerVolumeSpecName "kube-api-access-b2x9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:34.678213 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.678192    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-kube-api-access-j2j9h" pod "be9f9fb3-acd8-486c-a33c-51c0c8a8f225" (UID: "be9f9fb3-acd8-486c-a33c-51c0c8a8f225"). InnerVolumeSpecName "kube-api-access-j2j9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:34.776554 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776516    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-home\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-home\") pod \"725b6c25-dfeb-4b1a-85d2-794d81968f68\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") "
Nov 03 18:22:34.776752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776576    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-tmp-dir\") pod \"725b6c25-dfeb-4b1a-85d2-794d81968f68\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") "
Nov 03 18:22:34.776752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776616    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-dshm\") pod \"725b6c25-dfeb-4b1a-85d2-794d81968f68\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") "
Nov 03 18:22:34.776752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776696    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725b6c25-dfeb-4b1a-85d2-794d81968f68-kube-api-access-dn9d8\" (UniqueName: \"kubernetes.io/projected/725b6c25-dfeb-4b1a-85d2-794d81968f68-kube-api-access-dn9d8\") pod \"725b6c25-dfeb-4b1a-85d2-794d81968f68\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") "
Nov 03 18:22:34.776752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776737    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/725b6c25-dfeb-4b1a-85d2-794d81968f68-tls-certs\" (UniqueName: \"kubernetes.io/secret/725b6c25-dfeb-4b1a-85d2-794d81968f68-tls-certs\") pod \"725b6c25-dfeb-4b1a-85d2-794d81968f68\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") "
Nov 03 18:22:34.776984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776772    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-model-cache\") pod \"725b6c25-dfeb-4b1a-85d2-794d81968f68\" (UID: \"725b6c25-dfeb-4b1a-85d2-794d81968f68\") "
Nov 03 18:22:34.776984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776836    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-home" pod "725b6c25-dfeb-4b1a-85d2-794d81968f68" (UID: "725b6c25-dfeb-4b1a-85d2-794d81968f68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.776984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.776904    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-tmp-dir" pod "725b6c25-dfeb-4b1a-85d2-794d81968f68" (UID: "725b6c25-dfeb-4b1a-85d2-794d81968f68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.777109 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777082    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777109 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777101    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777115    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777130    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777144    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/383193f6-170f-4f03-a9f8-4fd37e2e11e1-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777157    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/383193f6-170f-4f03-a9f8-4fd37e2e11e1-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777169    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2x9r\" (UniqueName: \"kubernetes.io/projected/383193f6-170f-4f03-a9f8-4fd37e2e11e1-kube-api-access-b2x9r\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777193 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777184    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777198    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777126    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-model-cache" pod "725b6c25-dfeb-4b1a-85d2-794d81968f68" (UID: "725b6c25-dfeb-4b1a-85d2-794d81968f68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.777418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777214    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777230    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777243    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2j9h\" (UniqueName: \"kubernetes.io/projected/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-kube-api-access-j2j9h\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.777418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.777256    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be9f9fb3-acd8-486c-a33c-51c0c8a8f225-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.779000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.778979    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/725b6c25-dfeb-4b1a-85d2-794d81968f68-tls-certs" pod "725b6c25-dfeb-4b1a-85d2-794d81968f68" (UID: "725b6c25-dfeb-4b1a-85d2-794d81968f68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:22:34.779470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.779446    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-dshm" pod "725b6c25-dfeb-4b1a-85d2-794d81968f68" (UID: "725b6c25-dfeb-4b1a-85d2-794d81968f68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:34.779470 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.779463    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725b6c25-dfeb-4b1a-85d2-794d81968f68-kube-api-access-dn9d8" pod "725b6c25-dfeb-4b1a-85d2-794d81968f68" (UID: "725b6c25-dfeb-4b1a-85d2-794d81968f68"). InnerVolumeSpecName "kube-api-access-dn9d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:34.878178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.878103    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.878178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.878129    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9d8\" (UniqueName: \"kubernetes.io/projected/725b6c25-dfeb-4b1a-85d2-794d81968f68-kube-api-access-dn9d8\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.878178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.878141    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/725b6c25-dfeb-4b1a-85d2-794d81968f68-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:34.878178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:34.878150    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/725b6c25-dfeb-4b1a-85d2-794d81968f68-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:35.108033 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.107995    4493 generic.go:356] "Generic (PLEG): container finished" podID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerID="2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67" exitCode=0
Nov 03 18:22:35.108226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.108044    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"
Nov 03 18:22:35.108226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.108044    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" event={"ID":"be9f9fb3-acd8-486c-a33c-51c0c8a8f225","Type":"ContainerDied","Data":"2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67"}
Nov 03 18:22:35.108226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.108092    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c" event={"ID":"be9f9fb3-acd8-486c-a33c-51c0c8a8f225","Type":"ContainerDied","Data":"2be4292daab8898489e8d2ca369c63f2a0181357ebd19cd0b1f4ab61d44f48f0"}
Nov 03 18:22:35.108226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.108116    4493 scope.go:122] "RemoveContainer" containerID="2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67"
Nov 03 18:22:35.109746 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.109722    4493 generic.go:356] "Generic (PLEG): container finished" podID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerID="92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733" exitCode=0
Nov 03 18:22:35.109865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.109764    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" event={"ID":"383193f6-170f-4f03-a9f8-4fd37e2e11e1","Type":"ContainerDied","Data":"92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733"}
Nov 03 18:22:35.109865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.109788    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"
Nov 03 18:22:35.109865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.109799    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt" event={"ID":"383193f6-170f-4f03-a9f8-4fd37e2e11e1","Type":"ContainerDied","Data":"e04260554dfb8af1f87f488926529a084930d1c315aec5606e74edb47e18eed6"}
Nov 03 18:22:35.111449 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.111428    4493 generic.go:356] "Generic (PLEG): container finished" podID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerID="f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2" exitCode=0
Nov 03 18:22:35.111560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.111453    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" event={"ID":"725b6c25-dfeb-4b1a-85d2-794d81968f68","Type":"ContainerDied","Data":"f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2"}
Nov 03 18:22:35.111560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.111474    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"
Nov 03 18:22:35.111560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.111488    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j" event={"ID":"725b6c25-dfeb-4b1a-85d2-794d81968f68","Type":"ContainerDied","Data":"c49da14fd04bf5ff6e5182c70afa49b31fa2032975a854368ffd160f48dccff1"}
Nov 03 18:22:35.118075 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.118053    4493 scope.go:122] "RemoveContainer" containerID="2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67"
Nov 03 18:22:35.118391 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:35.118367    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67\": container with ID starting with 2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67 not found: ID does not exist" containerID="2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67"
Nov 03 18:22:35.118453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.118409    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67"} err="failed to get container status \"2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67\": rpc error: code = NotFound desc = could not find container \"2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67\": container with ID starting with 2c43c85694068da582a1d02e112dcb3b37e35d0068b0a1f6a75afc4e4c39ed67 not found: ID does not exist"
Nov 03 18:22:35.118453 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.118433    4493 scope.go:122] "RemoveContainer" containerID="92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733"
Nov 03 18:22:35.126640 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.126625    4493 scope.go:122] "RemoveContainer" containerID="92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733"
Nov 03 18:22:35.126981 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:35.126866    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733\": container with ID starting with 92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733 not found: ID does not exist" containerID="92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733"
Nov 03 18:22:35.127072 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.126990    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733"} err="failed to get container status \"92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733\": rpc error: code = NotFound desc = could not find container \"92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733\": container with ID starting with 92b176bafdf53365edc98ae1ba3808b3edd69c07147f481cd0b5ecf97859f733 not found: ID does not exist"
Nov 03 18:22:35.127072 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.127006    4493 scope.go:122] "RemoveContainer" containerID="f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2"
Nov 03 18:22:35.134673 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.134658    4493 scope.go:122] "RemoveContainer" containerID="f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2"
Nov 03 18:22:35.134984 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:35.134967    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2\": container with ID starting with f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2 not found: ID does not exist" containerID="f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2"
Nov 03 18:22:35.135027 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.134991    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2"} err="failed to get container status \"f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2\": rpc error: code = NotFound desc = could not find container \"f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2\": container with ID starting with f0c36a71b1ced322e2feded9568cca5650a3bfa8c27d8d6562ae841a6420b3e2 not found: ID does not exist"
Nov 03 18:22:35.278593 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.278556    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"]
Nov 03 18:22:35.287689 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.287661    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-w3-776ba7-kserve-9b6b5b568-nxhpt"]
Nov 03 18:22:35.402104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.402023    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"]
Nov 03 18:22:35.405103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.405074    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/distinct-2-w3-776ba7-kserve-68b8c6b8c5-rf78c"]
Nov 03 18:22:35.519176 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.519143    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"]
Nov 03 18:22:35.525104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:35.525074    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/simulator-w3-776ba7-kserve-f67d57f96-6rx6j"]
Nov 03 18:22:36.711168 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:36.711136    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" path="/var/lib/kubelet/pods/383193f6-170f-4f03-a9f8-4fd37e2e11e1/volumes"
Nov 03 18:22:36.711545 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:36.711514    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" path="/var/lib/kubelet/pods/725b6c25-dfeb-4b1a-85d2-794d81968f68/volumes"
Nov 03 18:22:36.711852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:36.711839    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" path="/var/lib/kubelet/pods/be9f9fb3-acd8-486c-a33c-51c0c8a8f225/volumes"
Nov 03 18:22:36.991568 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:36.991531    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:22:41.794400 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:41.794358    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" podUID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:41.794400 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:41.794377    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" podUID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 18:22:42.670729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:42.670696    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"]
Nov 03 18:22:42.721001 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:42.720971    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-776ba7-mxg5z"]
Nov 03 18:22:43.270579 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:43.270547    4493 scope.go:122] "RemoveContainer" containerID="ba4dfe3be35cd2901afeeb1f89b90626c8653e5cd0880d60fc4fafb0d7ace00b"
Nov 03 18:22:43.279211 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:43.279191    4493 scope.go:122] "RemoveContainer" containerID="04c2389093ce7d18b9933516bf03cb24ecc8217f8c1250cb224a952761867cfd"
Nov 03 18:22:43.286924 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:43.286904    4493 scope.go:122] "RemoveContainer" containerID="4695d3c859fb723cb584b1f71700d8469e2819154f5454a22e33363e7080fc64"
Nov 03 18:22:43.294328 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:43.294307    4493 scope.go:122] "RemoveContainer" containerID="85f2635b9d800c6627593e11c6f2e3161115526ce7b489a5649b53e3de78f9d5"
Nov 03 18:22:43.301731 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:43.301713    4493 scope.go:122] "RemoveContainer" containerID="7a544bd623dab4c2ed607ce1eea960e83bd32a002f33b201b05b4b08f567cc01"
Nov 03 18:22:43.309260 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:43.309237    4493 scope.go:122] "RemoveContainer" containerID="da32e40c7399124ac45d013c149b9e138655e58c73516b8d8ac6fa5b575abcbf"
Nov 03 18:22:44.710431 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:44.710397    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bd9871-a606-4da8-9f68-6daa52a25882" path="/var/lib/kubelet/pods/43bd9871-a606-4da8-9f68-6daa52a25882/volumes"
Nov 03 18:22:51.456169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.456135    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"]
Nov 03 18:22:51.456599 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.456351    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy" containerID="cri-o://d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944" gracePeriod=30
Nov 03 18:22:51.644716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.644646    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 18:22:51.743128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.743102    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 18:22:51.825198 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.825170    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264bf7be-e144-4ade-b8d2-e648ee3ba04f-kube-api-access-cpsnp\" (UniqueName: \"kubernetes.io/projected/264bf7be-e144-4ade-b8d2-e648ee3ba04f-kube-api-access-cpsnp\") pod \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") "
Nov 03 18:22:51.825372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.825220    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264bf7be-e144-4ade-b8d2-e648ee3ba04f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264bf7be-e144-4ade-b8d2-e648ee3ba04f-plugins-config-volume\") pod \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\" (UID: \"264bf7be-e144-4ade-b8d2-e648ee3ba04f\") "
Nov 03 18:22:51.825604 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.825580    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264bf7be-e144-4ade-b8d2-e648ee3ba04f-plugins-config-volume" pod "264bf7be-e144-4ade-b8d2-e648ee3ba04f" (UID: "264bf7be-e144-4ade-b8d2-e648ee3ba04f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:51.827394 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.827373    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264bf7be-e144-4ade-b8d2-e648ee3ba04f-kube-api-access-cpsnp" pod "264bf7be-e144-4ade-b8d2-e648ee3ba04f" (UID: "264bf7be-e144-4ade-b8d2-e648ee3ba04f"). InnerVolumeSpecName "kube-api-access-cpsnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:51.926608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.926536    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a38ae74-1b59-4b19-a578-1ebc71bd368e-kube-api-access-6vhtk\" (UniqueName: \"kubernetes.io/projected/0a38ae74-1b59-4b19-a578-1ebc71bd368e-kube-api-access-6vhtk\") pod \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") "
Nov 03 18:22:51.926760 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.926664    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a38ae74-1b59-4b19-a578-1ebc71bd368e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a38ae74-1b59-4b19-a578-1ebc71bd368e-plugins-config-volume\") pod \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\" (UID: \"0a38ae74-1b59-4b19-a578-1ebc71bd368e\") "
Nov 03 18:22:51.926812 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.926800    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpsnp\" (UniqueName: \"kubernetes.io/projected/264bf7be-e144-4ade-b8d2-e648ee3ba04f-kube-api-access-cpsnp\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:51.926847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.926815    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264bf7be-e144-4ade-b8d2-e648ee3ba04f-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:51.927057 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.927035    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a38ae74-1b59-4b19-a578-1ebc71bd368e-plugins-config-volume" pod "0a38ae74-1b59-4b19-a578-1ebc71bd368e" (UID: "0a38ae74-1b59-4b19-a578-1ebc71bd368e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:51.928793 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:51.928776    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a38ae74-1b59-4b19-a578-1ebc71bd368e-kube-api-access-6vhtk" pod "0a38ae74-1b59-4b19-a578-1ebc71bd368e" (UID: "0a38ae74-1b59-4b19-a578-1ebc71bd368e"). InnerVolumeSpecName "kube-api-access-6vhtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:52.027385 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.027352    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhtk\" (UniqueName: \"kubernetes.io/projected/0a38ae74-1b59-4b19-a578-1ebc71bd368e-kube-api-access-6vhtk\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:52.027385 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.027379    4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a38ae74-1b59-4b19-a578-1ebc71bd368e-plugins-config-volume\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:52.199680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.199591    4493 generic.go:356] "Generic (PLEG): container finished" podID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" containerID="2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c" exitCode=0
Nov 03 18:22:52.199680 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.199645    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"
Nov 03 18:22:52.199945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.199689    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" event={"ID":"264bf7be-e144-4ade-b8d2-e648ee3ba04f","Type":"ContainerDied","Data":"2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c"}
Nov 03 18:22:52.199945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.199747    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5" event={"ID":"264bf7be-e144-4ade-b8d2-e648ee3ba04f","Type":"ContainerDied","Data":"55b954e4a5bcd916c0878284eb06591cb72046cbb6c21e0ab6e3f662ae1597fd"}
Nov 03 18:22:52.199945 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.199778    4493 scope.go:122] "RemoveContainer" containerID="2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c"
Nov 03 18:22:52.201150 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.201119    4493 generic.go:356] "Generic (PLEG): container finished" podID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" containerID="8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83" exitCode=0
Nov 03 18:22:52.201261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.201175    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" event={"ID":"0a38ae74-1b59-4b19-a578-1ebc71bd368e","Type":"ContainerDied","Data":"8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83"}
Nov 03 18:22:52.201261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.201181    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"
Nov 03 18:22:52.201261 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.201209    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w" event={"ID":"0a38ae74-1b59-4b19-a578-1ebc71bd368e","Type":"ContainerDied","Data":"e31077d5aba2736cf116a91a59f50014bff646c19256cc17ddfe29293b0191f3"}
Nov 03 18:22:52.208507 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.208488    4493 scope.go:122] "RemoveContainer" containerID="2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c"
Nov 03 18:22:52.208789 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:52.208770    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c\": container with ID starting with 2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c not found: ID does not exist" containerID="2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c"
Nov 03 18:22:52.208892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.208800    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c"} err="failed to get container status \"2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c\": rpc error: code = NotFound desc = could not find container \"2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c\": container with ID starting with 2898814f59a1a335a4db261d29a9cc98320388a3a039a5a1d2a7d439c5ffac0c not found: ID does not exist"
Nov 03 18:22:52.208892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.208822    4493 scope.go:122] "RemoveContainer" containerID="8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83"
Nov 03 18:22:52.216833 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.216820    4493 scope.go:122] "RemoveContainer" containerID="8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83"
Nov 03 18:22:52.217109 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:52.217084    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83\": container with ID starting with 8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83 not found: ID does not exist" containerID="8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83"
Nov 03 18:22:52.217157 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.217116    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83"} err="failed to get container status \"8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83\": rpc error: code = NotFound desc = could not find container \"8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83\": container with ID starting with 8683692c85d21fb1ae8cfcda9f6c5631f25bc6bb140a2d0b0e3ef80667b0ba83 not found: ID does not exist"
Nov 03 18:22:52.371094 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.371060    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"]
Nov 03 18:22:52.374141 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.374114    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-776ba7-cd5d5c4bf-gvmt5"]
Nov 03 18:22:52.487215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.487183    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"]
Nov 03 18:22:52.490549 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.490519    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-776ba7-86dc6f6b47-2qf4w"]
Nov 03 18:22:52.710473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.710428    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" path="/var/lib/kubelet/pods/0a38ae74-1b59-4b19-a578-1ebc71bd368e/volumes"
Nov 03 18:22:52.710838 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:52.710825    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" path="/var/lib/kubelet/pods/264bf7be-e144-4ade-b8d2-e648ee3ba04f/volumes"
Nov 03 18:22:56.699513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.699485    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 18:22:56.877615 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877525    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-kube-api-access-h2fp5\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-kube-api-access-h2fp5\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877615 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877574    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-data\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877615 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877592    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc030dce-3454-4c6c-8ebb-e532e89676ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc030dce-3454-4c6c-8ebb-e532e89676ef-istiod-ca-cert\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877621    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-certs\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877651    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-podinfo\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877709    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-token\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-token\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877775    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-credential-socket\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877812    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-envoy\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.877946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877841    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-socket\") pod \"fc030dce-3454-4c6c-8ebb-e532e89676ef\" (UID: \"fc030dce-3454-4c6c-8ebb-e532e89676ef\") "
Nov 03 18:22:56.878289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877953    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-data" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:56.878289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.877984    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-certs" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:56.878289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.878067    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc030dce-3454-4c6c-8ebb-e532e89676ef-istiod-ca-cert" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:22:56.878289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.878170    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-socket" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:56.878289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.878255    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-credential-socket" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:56.878289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.878268    4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.878802 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.878385    4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-workload-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.878802 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.878469    4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-data\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.878802 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.878484    4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc030dce-3454-4c6c-8ebb-e532e89676ef-istiod-ca-cert\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.879856 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.879830    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-kube-api-access-h2fp5" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "kube-api-access-h2fp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:56.880025 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.880001    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-podinfo" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 18:22:56.880126 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.880077    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-token" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:22:56.880226 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.880208    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-envoy" pod "fc030dce-3454-4c6c-8ebb-e532e89676ef" (UID: "fc030dce-3454-4c6c-8ebb-e532e89676ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:22:56.979419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.979385    4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-credential-socket\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.979419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.979411    4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-envoy\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.979419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.979422    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2fp5\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-kube-api-access-h2fp5\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.979657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.979433    4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-podinfo\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:56.979657 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:56.979442    4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc030dce-3454-4c6c-8ebb-e532e89676ef-istio-token\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:22:57.226441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.226410    4493 generic.go:356] "Generic (PLEG): container finished" podID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerID="d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944" exitCode=0
Nov 03 18:22:57.226619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.226461    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"
Nov 03 18:22:57.226619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.226489    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" event={"ID":"fc030dce-3454-4c6c-8ebb-e532e89676ef","Type":"ContainerDied","Data":"d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944"}
Nov 03 18:22:57.226619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.226541    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf" event={"ID":"fc030dce-3454-4c6c-8ebb-e532e89676ef","Type":"ContainerDied","Data":"3ee7eb17d3a63daee094c49925f0fc8e74cb1df212ef5d8c5a15da6eab965c48"}
Nov 03 18:22:57.226619 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.226562    4493 scope.go:122] "RemoveContainer" containerID="d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944"
Nov 03 18:22:57.235742 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.235725    4493 scope.go:122] "RemoveContainer" containerID="d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944"
Nov 03 18:22:57.236036 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:22:57.236015    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944\": container with ID starting with d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944 not found: ID does not exist" containerID="d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944"
Nov 03 18:22:57.236089 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.236044    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944"} err="failed to get container status \"d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944\": rpc error: code = NotFound desc = could not find container \"d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944\": container with ID starting with d4fc4147921c332a9f224d097f646f195d25243be51cca5f66aadaf6061b1944 not found: ID does not exist"
Nov 03 18:22:57.396966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.396934    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"]
Nov 03 18:22:57.447674 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:57.447643    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-46kjf"]
Nov 03 18:22:58.710380 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:22:58.710352    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" path="/var/lib/kubelet/pods/fc030dce-3454-4c6c-8ebb-e532e89676ef/volumes"
Nov 03 18:23:06.991321 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:23:06.991277    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:23:20.218338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218301    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"]
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218594    4493 cpu_manager.go:414] "containerMap: removing container" podUID="288a7e72-ae62-46e0-80c6-b5d27c26edb1" containerName="revoke-keys"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218604    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288a7e72-ae62-46e0-80c6-b5d27c26edb1" containerName="revoke-keys"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218620    4493 cpu_manager.go:414] "containerMap: removing container" podUID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" containerName="payload-pre-processing"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218625    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" containerName="payload-pre-processing"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218632    4493 cpu_manager.go:414] "containerMap: removing container" podUID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" containerName="payload-processing"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218638    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" containerName="payload-processing"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218647    4493 cpu_manager.go:414] "containerMap: removing container" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218652    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218659    4493 cpu_manager.go:414] "containerMap: removing container" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218664    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218672    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218676    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218684    4493 cpu_manager.go:414] "containerMap: removing container" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218689    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218743    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="383193f6-170f-4f03-a9f8-4fd37e2e11e1" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218752    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9f9fb3-acd8-486c-a33c-51c0c8a8f225" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218757    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc030dce-3454-4c6c-8ebb-e532e89676ef" containerName="istio-proxy"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218763    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="725b6c25-dfeb-4b1a-85d2-794d81968f68" containerName="main"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218770    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="288a7e72-ae62-46e0-80c6-b5d27c26edb1" containerName="revoke-keys"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218776    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a38ae74-1b59-4b19-a578-1ebc71bd368e" containerName="payload-pre-processing"
Nov 03 18:23:20.218807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.218781    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="264bf7be-e144-4ade-b8d2-e648ee3ba04f" containerName="payload-processing"
Nov 03 18:23:20.221683 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.221665    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.268620 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.268592    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"]
Nov 03 18:23:20.271414 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271391    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271478 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271428    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271478 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271452    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb9ec24-1401-455e-b146-cd26986e90e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271552 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271509    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271570    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271638 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271623    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271660    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271685    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271720    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"]
Nov 03 18:23:20.271835 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271768    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzg67\" (UniqueName: \"kubernetes.io/projected/cfb9ec24-1401-455e-b146-cd26986e90e1-kube-api-access-rzg67\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.271914 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.271831    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.274722 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.274703    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.284017 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.283993    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 18:23:20.317571 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.317550    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqhn9\""
Nov 03 18:23:20.325108 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.325084    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"]
Nov 03 18:23:20.373045 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373014    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373045 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373043    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnf4s\" (UniqueName: \"kubernetes.io/projected/44987738-f4da-4721-93f0-c4abfa00e1b1-kube-api-access-hnf4s\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.373234 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373063    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373234 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373082    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.373234 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373205    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373234    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44987738-f4da-4721-93f0-c4abfa00e1b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.373356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373251    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.373356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373289    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373356 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373337    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373417    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373450    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373482    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373563    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtp8\" (UniqueName: \"kubernetes.io/projected/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-kube-api-access-6jtp8\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373626 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373585    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373709 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373624    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.373709 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373663    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb9ec24-1401-455e-b146-cd26986e90e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373709 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373698    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373716    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373734    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373766    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373791    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.373865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373821    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.373865 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373859    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg67\" (UniqueName: \"kubernetes.io/projected/cfb9ec24-1401-455e-b146-cd26986e90e1-kube-api-access-rzg67\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373907    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373933    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373939    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.373962    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.374033    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.374064    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.374109    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.374490    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb9ec24-1401-455e-b146-cd26986e90e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.374984 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.374617    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.377434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.377349    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.377434 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.377355    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"]
Nov 03 18:23:20.377606 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.377570    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.394448 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.394419    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"]
Nov 03 18:23:20.437672 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.437639    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb9ec24-1401-455e-b146-cd26986e90e1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.437781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.437752    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg67\" (UniqueName: \"kubernetes.io/projected/cfb9ec24-1401-455e-b146-cd26986e90e1-kube-api-access-rzg67\") pod \"maas-default-gateway-openshift-default-5df48fcc99-kz2rm\" (UID: \"cfb9ec24-1401-455e-b146-cd26986e90e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.475259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475199    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475230    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475247    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475265    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475285    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475300    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnf4s\" (UniqueName: \"kubernetes.io/projected/44987738-f4da-4721-93f0-c4abfa00e1b1-kube-api-access-hnf4s\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475318    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475332    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475360    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475375    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44987738-f4da-4721-93f0-c4abfa00e1b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475390    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475415    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475430    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475447    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475464    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475489    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtp8\" (UniqueName: \"kubernetes.io/projected/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-kube-api-access-6jtp8\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475512    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.475754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.475550    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.476283 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.476260    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.476430 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.476404    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.476527 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.476463    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.476726 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.476702    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.476996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.476854    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.476996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.476891    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.476996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.476967    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.477200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.477145    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.477200 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.477171    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.477547 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.477529    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44987738-f4da-4721-93f0-c4abfa00e1b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.478592 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.478564    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.478988 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.478963    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.479516 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.479493    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.480395 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.480369    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.533496 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.533475    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:20.539306 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.539283    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.539409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.539326    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44987738-f4da-4721-93f0-c4abfa00e1b1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.539735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.539717    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnf4s\" (UniqueName: \"kubernetes.io/projected/44987738-f4da-4721-93f0-c4abfa00e1b1-kube-api-access-hnf4s\") pod \"maas-default-gateway-openshift-default-5df48fcc99-6s5qj\" (UID: \"44987738-f4da-4721-93f0-c4abfa00e1b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.540426 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.540410    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtp8\" (UniqueName: \"kubernetes.io/projected/7daacb4d-0cfe-42f6-bb51-6e08c57ddced-kube-api-access-6jtp8\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zcfbk\" (UID: \"7daacb4d-0cfe-42f6-bb51-6e08c57ddced\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.585803 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.585128    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:20.591041 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.591014    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:20.962862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.962841    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"]
Nov 03 18:23:20.965003 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:23:20.964980    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb9ec24_1401_455e_b146_cd26986e90e1.slice/crio-bec599d17b6cdbb68742f69f8e989a56b6b216e9d6f3cdab2fcad64764aff661 WatchSource:0}: Error finding container bec599d17b6cdbb68742f69f8e989a56b6b216e9d6f3cdab2fcad64764aff661: Status 404 returned error can't find the container with id bec599d17b6cdbb68742f69f8e989a56b6b216e9d6f3cdab2fcad64764aff661
Nov 03 18:23:20.967100 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.967066    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:20.967184 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.967130    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:20.967184 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:20.967162    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:21.017281 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.017201    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"]
Nov 03 18:23:21.018918 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:23:21.018892    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7daacb4d_0cfe_42f6_bb51_6e08c57ddced.slice/crio-d65372b95f8d9b13c3699b6b8f47e10f367bb2914a1046d3aaddcf23310c40a4 WatchSource:0}: Error finding container d65372b95f8d9b13c3699b6b8f47e10f367bb2914a1046d3aaddcf23310c40a4: Status 404 returned error can't find the container with id d65372b95f8d9b13c3699b6b8f47e10f367bb2914a1046d3aaddcf23310c40a4
Nov 03 18:23:21.021206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.021169    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:21.021290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.021232    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:21.021290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.021258    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:21.067528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.067497    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"]
Nov 03 18:23:21.073091 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:23:21.073056    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44987738_f4da_4721_93f0_c4abfa00e1b1.slice/crio-a32a265d9729c08d91573a35a5e37b78674290f4ef414113b875aa0f37ae310e WatchSource:0}: Error finding container a32a265d9729c08d91573a35a5e37b78674290f4ef414113b875aa0f37ae310e: Status 404 returned error can't find the container with id a32a265d9729c08d91573a35a5e37b78674290f4ef414113b875aa0f37ae310e
Nov 03 18:23:21.074762 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.074731    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:21.074846 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.074795    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:21.074846 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.074824    4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 18:23:21.344763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.344365    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" event={"ID":"7daacb4d-0cfe-42f6-bb51-6e08c57ddced","Type":"ContainerStarted","Data":"d78d9aef0031d86a735ca43b6ea810619dde1f5b182d5e5c8530b0ab8b1f4fdd"}
Nov 03 18:23:21.344763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.344429    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" event={"ID":"7daacb4d-0cfe-42f6-bb51-6e08c57ddced","Type":"ContainerStarted","Data":"d65372b95f8d9b13c3699b6b8f47e10f367bb2914a1046d3aaddcf23310c40a4"}
Nov 03 18:23:21.346710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.346474    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" event={"ID":"44987738-f4da-4721-93f0-c4abfa00e1b1","Type":"ContainerStarted","Data":"4eb5ff1dcbdb543ff7bf6102eb3c651a7ffbcd16bfbf6c6b54bc4e3dc9ea7274"}
Nov 03 18:23:21.346710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.346531    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" event={"ID":"44987738-f4da-4721-93f0-c4abfa00e1b1","Type":"ContainerStarted","Data":"a32a265d9729c08d91573a35a5e37b78674290f4ef414113b875aa0f37ae310e"}
Nov 03 18:23:21.348660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.348188    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" event={"ID":"cfb9ec24-1401-455e-b146-cd26986e90e1","Type":"ContainerStarted","Data":"4ae292c95f8e2f9acd2850d4dc049cf331298a9108be7e79b146cfe072479f57"}
Nov 03 18:23:21.348660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.348231    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" event={"ID":"cfb9ec24-1401-455e-b146-cd26986e90e1","Type":"ContainerStarted","Data":"bec599d17b6cdbb68742f69f8e989a56b6b216e9d6f3cdab2fcad64764aff661"}
Nov 03 18:23:21.470983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.470925    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podStartSLOduration=1.470908533 podStartE2EDuration="1.470908533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:21.469251174 +0000 UTC m=+2980.928422161" watchObservedRunningTime="2026-11-03 18:23:21.470908533 +0000 UTC m=+2980.930079518"
Nov 03 18:23:21.534111 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.534073    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:21.585587 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.585549    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:21.591811 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.591787    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:21.596694 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.596608    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podStartSLOduration=1.596588098 podStartE2EDuration="1.596588098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:21.595069834 +0000 UTC m=+2981.054240821" watchObservedRunningTime="2026-11-03 18:23:21.596588098 +0000 UTC m=+2981.055759124"
Nov 03 18:23:21.721498 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:21.721435    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podStartSLOduration=1.721414885 podStartE2EDuration="1.721414885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:21.720061789 +0000 UTC m=+2981.179232778" watchObservedRunningTime="2026-11-03 18:23:21.721414885 +0000 UTC m=+2981.180585872"
Nov 03 18:23:22.539847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:22.539806    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 18:23:22.540515 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:22.540478    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 18:23:22.598206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:22.598167    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 18:23:22.598611 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:22.598581    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 18:23:22.598807 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:22.598491    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 18:23:22.598954 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:22.598934    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 18:23:23.537432 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:23.537356    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 18:23:23.537672 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:23.537439    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 18:23:23.589510 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:23.589470    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 18:23:23.590912 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:23.590111    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 18:23:23.592525 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:23.592479    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 18:23:23.592663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:23.592551    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 18:23:24.536899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:24.536291    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 18:23:24.536899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:24.536368    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 18:23:24.586899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:24.586449    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 18:23:24.586899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:24.586526    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 18:23:24.592310 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:24.592084    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 18:23:24.592310 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:24.592138    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 18:23:25.537377 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:25.537325    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 18:23:25.537701 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:25.537670    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 18:23:25.587405 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:25.587362    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 18:23:25.587698 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:25.587665    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 18:23:25.592366 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:25.592339    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 18:23:25.592754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:25.592732    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 18:23:26.536393 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:26.536353    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 18:23:26.536710 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:26.536682    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 18:23:26.587370 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:26.587329    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 18:23:26.587628 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:26.587601    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 18:23:26.592347 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:26.592315    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 18:23:26.592503 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:26.592374    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 18:23:27.534145 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:27.534107    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 18:23:27.534316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:27.534180    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 18:23:27.586776 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:27.586735    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 18:23:27.586950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:27.586803    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 18:23:27.591836 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:27.591801    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 18:23:27.592021 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:27.591860    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 18:23:28.534970 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:28.534929    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 18:23:28.535407 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:28.535003    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 18:23:28.586588 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:28.586549    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 18:23:28.586768 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:28.586617    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 18:23:28.591998 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:28.591967    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Nov 03 18:23:28.592131 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:28.592026    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Nov 03 18:23:30.576064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:30.576017    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:23:30.576602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:30.576087    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:23:30.586860 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:30.586817    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:23:30.587040 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:30.586896    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:23:30.592672 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:30.592644    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:23:30.592775 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:30.592698    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:23:31.620110 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:31.620032    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:23:31.620815 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:31.620772    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:23:31.634077 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:31.634020    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:23:31.634287 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:31.634260    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:23:31.634852 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:31.634830    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:23:31.635061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:31.635038    4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:23:32.284289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.284249    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:32.285231 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.285212    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:32.286076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.286049    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm"
Nov 03 18:23:32.428482 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.428433    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:32.428931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.428905    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:32.429615 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.429594    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk"
Nov 03 18:23:32.447061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.447034    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:32.447390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.447367    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:32.516203 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:32.516180    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj"
Nov 03 18:23:36.990958 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:23:36.990920    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:23:43.370570 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:43.370538    4493 scope.go:122] "RemoveContainer" containerID="7dc702d6b09fc9e36ccb0c4f5c0775ac3375f92ebe2a80d1a3b2815da3c1f6e7"
Nov 03 18:23:43.378921 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:43.378900    4493 scope.go:122] "RemoveContainer" containerID="8ff9ba2e75cb2b940c8326407d418e8e28242e75331a21b4a5d7e084a61b706b"
Nov 03 18:23:43.386171 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:23:43.386153    4493 scope.go:122] "RemoveContainer" containerID="02ede4170ef46fe3275990f18b2537a90f7e40cbe28174b24389ead1cf749370"
Nov 03 18:24:06.994445 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:24:06.994359    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:24:36.991320 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:24:36.991278    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:24:43.420731 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:24:43.420698    4493 scope.go:122] "RemoveContainer" containerID="18f77614a35c2f3a3af7d59e6b68829c5e5fa96de8306c3981cc1bb9469719aa"
Nov 03 18:24:43.428973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:24:43.428942    4493 scope.go:122] "RemoveContainer" containerID="bc22a51dc9cd908b23312b6fc45bd02850a3f8cc25985f44a7fb27bd1724d8e3"
Nov 03 18:25:06.992272 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:25:06.992230    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:25:20.057586 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:20.057546    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"]
Nov 03 18:25:20.108452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:20.108414    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05af8a05-snq8t"]
Nov 03 18:25:20.710973 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:20.710943    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deee988e-fdea-46c6-9b4f-ff9582203d4b" path="/var/lib/kubelet/pods/deee988e-fdea-46c6-9b4f-ff9582203d4b/volumes"
Nov 03 18:25:29.056154 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:29.056119    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"]
Nov 03 18:25:29.106641 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:29.106609    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fa601de-8w2zs"]
Nov 03 18:25:30.058173 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:30.058133    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"]
Nov 03 18:25:30.108637 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:30.108606    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7bb928-hdv4d"]
Nov 03 18:25:30.710312 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:30.710274    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0448e549-eef3-4706-81ac-f26c44eca459" path="/var/lib/kubelet/pods/0448e549-eef3-4706-81ac-f26c44eca459/volumes"
Nov 03 18:25:30.710679 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:30.710665    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eebc30cd-e785-4a2a-88ed-1f6143bc0bdd" path="/var/lib/kubelet/pods/eebc30cd-e785-4a2a-88ed-1f6143bc0bdd/volumes"
Nov 03 18:25:33.058716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:33.058682    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"]
Nov 03 18:25:33.109332 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:33.109299    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9ffb86-q5846"]
Nov 03 18:25:34.712788 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:34.712735    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e66a96-1f1f-4e19-90eb-3f6c3b5e569a" path="/var/lib/kubelet/pods/63e66a96-1f1f-4e19-90eb-3f6c3b5e569a/volumes"
Nov 03 18:25:36.999131 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:25:36.999080    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:25:43.449331 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:25:43.449299    4493 scope.go:122] "RemoveContainer" containerID="da1490e3f10ab95d41e03691cc531612c77ece059b6b168446151f3a0c54ee40"
Nov 03 18:26:07.007270 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:26:07.007233    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:26:07.064668 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:07.064629    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"]
Nov 03 18:26:07.113838 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:07.113807    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19710e6d-7zrcj"]
Nov 03 18:26:08.714066 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:08.711817    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb97c6b-8697-4a9b-af7a-f5d760e26543" path="/var/lib/kubelet/pods/7eb97c6b-8697-4a9b-af7a-f5d760e26543/volumes"
Nov 03 18:26:36.991626 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:26:36.991590    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:26:43.468731 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:43.468697    4493 scope.go:122] "RemoveContainer" containerID="28f1b5df2a1ccf813a4cbb970ee35a7b67264cd86cf8ea2efda50c550ad24e87"
Nov 03 18:26:43.478373 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:43.478352    4493 scope.go:122] "RemoveContainer" containerID="e98bafb5db56d65f85315f07b1797328b92d0bf5634a148c8fc5980a3cd78649"
Nov 03 18:26:43.486046 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:43.486027    4493 scope.go:122] "RemoveContainer" containerID="a4223d8a40916a8f9e55fec9e979d893057ab8ae58abab45a378aec9657a6678"
Nov 03 18:26:43.493339 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:43.493317    4493 scope.go:122] "RemoveContainer" containerID="8cb47b28168b58118df98e42f98fd32d73de0a5ab28c7665590cc57bbad7e424"
Nov 03 18:26:43.500763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:43.500742    4493 scope.go:122] "RemoveContainer" containerID="8cada398515d4d43e78187077333b1e22a455203110e7c21b249c8597f2c03ea"
Nov 03 18:26:43.508317 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:26:43.508300    4493 scope.go:122] "RemoveContainer" containerID="6738bf90cedc39723bbc1cd57b8fdf316c72f2599a6f7002e90cc3e97c4f6697"
Nov 03 18:27:06.991984 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:27:06.991907    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:27:30.058068 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:27:30.058031    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"]
Nov 03 18:27:30.100820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:27:30.100788    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7373309d-kwfxp"]
Nov 03 18:27:30.710480 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:27:30.710447    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288a7e72-ae62-46e0-80c6-b5d27c26edb1" path="/var/lib/kubelet/pods/288a7e72-ae62-46e0-80c6-b5d27c26edb1/volumes"
Nov 03 18:27:36.990641 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:27:36.990601    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:27:43.541767 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:27:43.541727    4493 scope.go:122] "RemoveContainer" containerID="bbd5304ec5d8e804df5d97003ac765069907da801d8b775068278af01694c109"
Nov 03 18:27:43.549820 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:27:43.549801    4493 scope.go:122] "RemoveContainer" containerID="6f75c0aaa3b1a435a790eed2e4b61ea881650caa9672bf1c0fe0804f8ac81740"
Nov 03 18:28:03.333373 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:03.332982    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:28:03.333373 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:03.333065    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:28:03.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:03.476984    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:28:03.477899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:03.477047    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:28:03.566899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:03.561975    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:28:03.566899 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:03.562036    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:28:06.992246 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:28:06.992208    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:28:36.992155 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:28:36.992109    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:28:43.580902 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:43.578022    4493 scope.go:122] "RemoveContainer" containerID="c54324338f041020825ff39a2714b79f0c1fba37d00c28d2b06a4e040c9c7349"
Nov 03 18:28:43.596271 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:43.596248    4493 scope.go:122] "RemoveContainer" containerID="4a9edd04cea2ab24f3589fe4442d71b8533580e743a41f3a82870e904fb9da70"
Nov 03 18:28:43.603799 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:28:43.603776    4493 scope.go:122] "RemoveContainer" containerID="e9c5c36e90a15253bf837d8eb62c07ea71340609372ec7c15f5e9476b8855890"
Nov 03 18:29:06.991835 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:29:06.991797    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:29:37.009595 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:29:37.009559    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:30:06.992888 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:30:06.992783    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:30:10.771127 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.771097    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"]
Nov 03 18:30:10.774475 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.774453    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:10.828057 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.828026    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:30:10.878655 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.878616    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:30:10.878655 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.878647    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:30:10.878867 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.878678    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\""
Nov 03 18:30:10.880110 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.880092    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"]
Nov 03 18:30:10.915815 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.915782    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:10.915815 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.915817    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:10.916035 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:10.915924    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46v7\" (UniqueName: \"kubernetes.io/projected/cf03b94c-13a1-44ca-850d-5171f331b55a-kube-api-access-r46v7\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.016735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.016701    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.016735 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.016738    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.017042 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.016774    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46v7\" (UniqueName: \"kubernetes.io/projected/cf03b94c-13a1-44ca-850d-5171f331b55a-kube-api-access-r46v7\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.017402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.017375    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.019198 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.019176    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.080418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.080353    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46v7\" (UniqueName: \"kubernetes.io/projected/cf03b94c-13a1-44ca-850d-5171f331b55a-kube-api-access-r46v7\") pod \"maas-api-revoke-sub-3d4156f1-8qsf7\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.087215 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.087193    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:11.462450 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:11.462424    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"]
Nov 03 18:30:11.465064 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:30:11.465026    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf03b94c_13a1_44ca_850d_5171f331b55a.slice/crio-ea26785844878bb8c2910d7ca1aeb1009dc16a506f8bd4a308e05abb3f17147e WatchSource:0}: Error finding container ea26785844878bb8c2910d7ca1aeb1009dc16a506f8bd4a308e05abb3f17147e: Status 404 returned error can't find the container with id ea26785844878bb8c2910d7ca1aeb1009dc16a506f8bd4a308e05abb3f17147e
Nov 03 18:30:12.353212 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:12.353181    4493 generic.go:356] "Generic (PLEG): container finished" podID="cf03b94c-13a1-44ca-850d-5171f331b55a" containerID="5b3f12ab3117046333c15037ed0311243073dd2a83e9809b78e56aa896ab9730" exitCode=0
Nov 03 18:30:12.353583 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:12.353271    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7" event={"ID":"cf03b94c-13a1-44ca-850d-5171f331b55a","Type":"ContainerDied","Data":"5b3f12ab3117046333c15037ed0311243073dd2a83e9809b78e56aa896ab9730"}
Nov 03 18:30:12.353583 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:12.353326    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7" event={"ID":"cf03b94c-13a1-44ca-850d-5171f331b55a","Type":"ContainerStarted","Data":"ea26785844878bb8c2910d7ca1aeb1009dc16a506f8bd4a308e05abb3f17147e"}
Nov 03 18:30:13.480388 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.480366    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:13.642526 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.642453    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-openshift-ca\") pod \"cf03b94c-13a1-44ca-850d-5171f331b55a\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") "
Nov 03 18:30:13.642526 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.642491    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf03b94c-13a1-44ca-850d-5171f331b55a-kube-api-access-r46v7\" (UniqueName: \"kubernetes.io/projected/cf03b94c-13a1-44ca-850d-5171f331b55a-kube-api-access-r46v7\") pod \"cf03b94c-13a1-44ca-850d-5171f331b55a\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") "
Nov 03 18:30:13.642716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.642580    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-xks-ca\") pod \"cf03b94c-13a1-44ca-850d-5171f331b55a\" (UID: \"cf03b94c-13a1-44ca-850d-5171f331b55a\") "
Nov 03 18:30:13.642908 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.642864    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-openshift-ca" pod "cf03b94c-13a1-44ca-850d-5171f331b55a" (UID: "cf03b94c-13a1-44ca-850d-5171f331b55a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:30:13.644781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.644752    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-xks-ca" pod "cf03b94c-13a1-44ca-850d-5171f331b55a" (UID: "cf03b94c-13a1-44ca-850d-5171f331b55a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:30:13.644781 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.644774    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf03b94c-13a1-44ca-850d-5171f331b55a-kube-api-access-r46v7" pod "cf03b94c-13a1-44ca-850d-5171f331b55a" (UID: "cf03b94c-13a1-44ca-850d-5171f331b55a"). InnerVolumeSpecName "kube-api-access-r46v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:30:13.743754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.743727    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:30:13.743754 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.743750    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46v7\" (UniqueName: \"kubernetes.io/projected/cf03b94c-13a1-44ca-850d-5171f331b55a-kube-api-access-r46v7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:30:13.743947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:13.743762    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf03b94c-13a1-44ca-850d-5171f331b55a-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:30:14.362263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:14.362226    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7" event={"ID":"cf03b94c-13a1-44ca-850d-5171f331b55a","Type":"ContainerDied","Data":"ea26785844878bb8c2910d7ca1aeb1009dc16a506f8bd4a308e05abb3f17147e"}
Nov 03 18:30:14.362263 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:14.362265    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"
Nov 03 18:30:14.362539 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:30:14.362275    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea26785844878bb8c2910d7ca1aeb1009dc16a506f8bd4a308e05abb3f17147e"
Nov 03 18:30:36.991617 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:30:36.991577    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:31:06.991376 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:31:06.991337    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:31:16.036381 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.036346    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"]
Nov 03 18:31:16.036825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.036681    4493 cpu_manager.go:414] "containerMap: removing container" podUID="cf03b94c-13a1-44ca-850d-5171f331b55a" containerName="revoke-keys"
Nov 03 18:31:16.036825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.036692    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf03b94c-13a1-44ca-850d-5171f331b55a" containerName="revoke-keys"
Nov 03 18:31:16.036825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.036751    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf03b94c-13a1-44ca-850d-5171f331b55a" containerName="revoke-keys"
Nov 03 18:31:16.039576 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.039558    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.113618 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.113587    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ca6dbf\"/\"kube-root-ca.crt\""
Nov 03 18:31:16.118195 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.118168    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4g7\" (UniqueName: \"kubernetes.io/projected/014d788f-1049-4c6c-b270-b78b56c42125-kube-api-access-4v4g7\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.118333 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.118259    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-tmp-dir\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.118333 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.118283    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014d788f-1049-4c6c-b270-b78b56c42125-tls-certs\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.118426 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.118341    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-home\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.118477 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.118456    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-model-cache\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.118555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.118541    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-dshm\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.144093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.144061    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ca6dbf\"/\"openshift-service-ca.crt\""
Nov 03 18:31:16.144222 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.144105    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ca6dbf\"/\"default-dockercfg-cmvb9\""
Nov 03 18:31:16.144472 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.144457    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ca6dbf\"/\"simulator-single-main-ca6dbf-kserve-self-signed-certs\""
Nov 03 18:31:16.146560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.146542    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"]
Nov 03 18:31:16.219374 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219343    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-dshm\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219382    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4g7\" (UniqueName: \"kubernetes.io/projected/014d788f-1049-4c6c-b270-b78b56c42125-kube-api-access-4v4g7\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219413    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-tmp-dir\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219429    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014d788f-1049-4c6c-b270-b78b56c42125-tls-certs\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219552    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-home\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219636    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-model-cache\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219825 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219807    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-tmp-dir\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219931    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-home\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.219974 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.219962    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-model-cache\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.221795 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.221777    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-dshm\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.222076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.222059    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014d788f-1049-4c6c-b270-b78b56c42125-tls-certs\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.283559 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.283521    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4g7\" (UniqueName: \"kubernetes.io/projected/014d788f-1049-4c6c-b270-b78b56c42125-kube-api-access-4v4g7\") pod \"simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") " pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.350513 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.350431    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:16.730411 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:16.730389    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"]
Nov 03 18:31:16.732937 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:31:16.732855    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod014d788f_1049_4c6c_b270_b78b56c42125.slice/crio-4b95928551f323c70acf25dcd3d8a67a4d8e21bdeb0c77343efab95d7a21b414 WatchSource:0}: Error finding container 4b95928551f323c70acf25dcd3d8a67a4d8e21bdeb0c77343efab95d7a21b414: Status 404 returned error can't find the container with id 4b95928551f323c70acf25dcd3d8a67a4d8e21bdeb0c77343efab95d7a21b414
Nov 03 18:31:17.619981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:17.619945    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl" event={"ID":"014d788f-1049-4c6c-b270-b78b56c42125","Type":"ContainerStarted","Data":"ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa"}
Nov 03 18:31:17.620379 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:17.620004    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl" event={"ID":"014d788f-1049-4c6c-b270-b78b56c42125","Type":"ContainerStarted","Data":"4b95928551f323c70acf25dcd3d8a67a4d8e21bdeb0c77343efab95d7a21b414"}
Nov 03 18:31:17.740833 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:17.740780    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl" podStartSLOduration=2.740765682 podStartE2EDuration="2.740765682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:31:17.740021549 +0000 UTC m=+3457.199192567" watchObservedRunningTime="2026-11-03 18:31:17.740765682 +0000 UTC m=+3457.199936659"
Nov 03 18:31:18.351169 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:18.351138    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:26.351358 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:26.351279    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:26.364249 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:26.364224    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:26.669635 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:31:26.669550    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:31:36.991370 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:31:36.991334    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:32:06.991616 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:32:06.991580    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:32:17.953063 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:17.953027    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"]
Nov 03 18:32:32.972714 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:32.972638    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl" podUID="014d788f-1049-4c6c-b270-b78b56c42125" containerName="main" containerID="cri-o://ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa" gracePeriod=45
Nov 03 18:32:33.222950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.222893    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:32:33.315792 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.315763    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-model-cache\") pod \"014d788f-1049-4c6c-b270-b78b56c42125\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") "
Nov 03 18:32:33.315975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.315830    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014d788f-1049-4c6c-b270-b78b56c42125-tls-certs\" (UniqueName: \"kubernetes.io/secret/014d788f-1049-4c6c-b270-b78b56c42125-tls-certs\") pod \"014d788f-1049-4c6c-b270-b78b56c42125\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") "
Nov 03 18:32:33.315975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.315856    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-tmp-dir\") pod \"014d788f-1049-4c6c-b270-b78b56c42125\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") "
Nov 03 18:32:33.315975 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.315902    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-dshm\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-dshm\") pod \"014d788f-1049-4c6c-b270-b78b56c42125\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") "
Nov 03 18:32:33.316163 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.316090    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-model-cache" pod "014d788f-1049-4c6c-b270-b78b56c42125" (UID: "014d788f-1049-4c6c-b270-b78b56c42125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:32:33.316163 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.316099    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-tmp-dir" pod "014d788f-1049-4c6c-b270-b78b56c42125" (UID: "014d788f-1049-4c6c-b270-b78b56c42125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:32:33.318206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.318170    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014d788f-1049-4c6c-b270-b78b56c42125-tls-certs" pod "014d788f-1049-4c6c-b270-b78b56c42125" (UID: "014d788f-1049-4c6c-b270-b78b56c42125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:32:33.318206 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.318187    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-dshm" pod "014d788f-1049-4c6c-b270-b78b56c42125" (UID: "014d788f-1049-4c6c-b270-b78b56c42125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:32:33.416564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.416532    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-home\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-home\") pod \"014d788f-1049-4c6c-b270-b78b56c42125\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") "
Nov 03 18:32:33.416736 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.416571    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014d788f-1049-4c6c-b270-b78b56c42125-kube-api-access-4v4g7\" (UniqueName: \"kubernetes.io/projected/014d788f-1049-4c6c-b270-b78b56c42125-kube-api-access-4v4g7\") pod \"014d788f-1049-4c6c-b270-b78b56c42125\" (UID: \"014d788f-1049-4c6c-b270-b78b56c42125\") "
Nov 03 18:32:33.416826 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.416807    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/014d788f-1049-4c6c-b270-b78b56c42125-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:33.416892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.416821    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-home" pod "014d788f-1049-4c6c-b270-b78b56c42125" (UID: "014d788f-1049-4c6c-b270-b78b56c42125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:32:33.416892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.416832    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:33.416892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.416845    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:33.416892 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.416855    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:33.418896 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.418843    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014d788f-1049-4c6c-b270-b78b56c42125-kube-api-access-4v4g7" pod "014d788f-1049-4c6c-b270-b78b56c42125" (UID: "014d788f-1049-4c6c-b270-b78b56c42125"). InnerVolumeSpecName "kube-api-access-4v4g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:32:33.517416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.517325    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/014d788f-1049-4c6c-b270-b78b56c42125-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:33.517416 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.517356    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v4g7\" (UniqueName: \"kubernetes.io/projected/014d788f-1049-4c6c-b270-b78b56c42125-kube-api-access-4v4g7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:32:33.927179 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.927104    4493 generic.go:356] "Generic (PLEG): container finished" podID="014d788f-1049-4c6c-b270-b78b56c42125" containerID="ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa" exitCode=0
Nov 03 18:32:33.927179 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.927150    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"
Nov 03 18:32:33.927368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.927187    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl" event={"ID":"014d788f-1049-4c6c-b270-b78b56c42125","Type":"ContainerDied","Data":"ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa"}
Nov 03 18:32:33.927368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.927233    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl" event={"ID":"014d788f-1049-4c6c-b270-b78b56c42125","Type":"ContainerDied","Data":"4b95928551f323c70acf25dcd3d8a67a4d8e21bdeb0c77343efab95d7a21b414"}
Nov 03 18:32:33.927368 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.927251    4493 scope.go:122] "RemoveContainer" containerID="ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa"
Nov 03 18:32:33.935803 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.935787    4493 scope.go:122] "RemoveContainer" containerID="ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa"
Nov 03 18:32:33.936110 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:32:33.936092    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa\": container with ID starting with ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa not found: ID does not exist" containerID="ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa"
Nov 03 18:32:33.936172 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:33.936118    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa"} err="failed to get container status \"ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa\": rpc error: code = NotFound desc = could not find container \"ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa\": container with ID starting with ed82bc42ab1ed14714e17e17c2481c5b59295a7052262b1aed7fa7ba9782b1fa not found: ID does not exist"
Nov 03 18:32:34.096864 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:34.096834    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"]
Nov 03 18:32:34.102700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:34.102677    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ca6dbf/simulator-single-main-ca6dbf-kserve-74c8dc4d56-qzbrl"]
Nov 03 18:32:34.710629 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:34.710593    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014d788f-1049-4c6c-b270-b78b56c42125" path="/var/lib/kubelet/pods/014d788f-1049-4c6c-b270-b78b56c42125/volumes"
Nov 03 18:32:36.996500 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:32:36.996459    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:32:39.778535 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:39.778505    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"]
Nov 03 18:32:39.829012 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:39.828981    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8qsf7"]
Nov 03 18:32:40.710946 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:32:40.710916    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf03b94c-13a1-44ca-850d-5171f331b55a" path="/var/lib/kubelet/pods/cf03b94c-13a1-44ca-850d-5171f331b55a/volumes"
Nov 03 18:33:06.991333 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:33:06.991296    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:33:36.991465 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:33:36.991426    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:34:06.991810 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:34:06.991766    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:34:36.991133 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:34:36.991096    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:35:06.990485 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:35:06.990441    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:35:36.991572 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:35:36.991535    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:36:02.041891 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.041803    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"]
Nov 03 18:36:02.042258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.042173    4493 cpu_manager.go:414] "containerMap: removing container" podUID="014d788f-1049-4c6c-b270-b78b56c42125" containerName="main"
Nov 03 18:36:02.042258 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.042189    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014d788f-1049-4c6c-b270-b78b56c42125" containerName="main"
Nov 03 18:36:02.042332 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.042273    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="014d788f-1049-4c6c-b270-b78b56c42125" containerName="main"
Nov 03 18:36:02.045073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.045057    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.101981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.101950    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-057118\"/\"default-dockercfg-xcxs6\""
Nov 03 18:36:02.105786 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.105759    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-model-cache\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.105949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.105802    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6z6\" (UniqueName: \"kubernetes.io/projected/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-kube-api-access-qq6z6\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.105949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.105824    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-home\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.106048 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.105974    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tmp-dir\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.106048 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.106006    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-dshm\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.106133 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.106064    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tls-certs\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.152736 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.152707    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-057118\"/\"kube-root-ca.crt\""
Nov 03 18:36:02.152736 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.152711    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-057118\"/\"simulator-single-main-057118-kserve-self-signed-certs\""
Nov 03 18:36:02.153916 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.153896    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-057118\"/\"openshift-service-ca.crt\""
Nov 03 18:36:02.154095 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.154078    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"]
Nov 03 18:36:02.206866 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.206829    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tmp-dir\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.206866 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.206867    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-dshm\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.207185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.206922    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tls-certs\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.207185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.206992    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-model-cache\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.207185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.207032    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6z6\" (UniqueName: \"kubernetes.io/projected/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-kube-api-access-qq6z6\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.207185 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.207062    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-home\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.207419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.207323    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tmp-dir\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.207419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.207401    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-model-cache\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.207539 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.207413    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-home\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.209515 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.209490    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-dshm\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.209652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.209606    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tls-certs\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.270571 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.270532    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6z6\" (UniqueName: \"kubernetes.io/projected/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-kube-api-access-qq6z6\") pod \"simulator-single-main-057118-kserve-57d5f74c76-vngk7\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") " pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.356683 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.356602    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:02.740428 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.740394    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"]
Nov 03 18:36:02.743362 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:36:02.743338    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod073e1231_6e9f_481b_9f9d_fbd01d8b2c33.slice/crio-cab90450dd22c788300244da2366ef55d090184ac75ee0d54c800a99d9d4bf0f WatchSource:0}: Error finding container cab90450dd22c788300244da2366ef55d090184ac75ee0d54c800a99d9d4bf0f: Status 404 returned error can't find the container with id cab90450dd22c788300244da2366ef55d090184ac75ee0d54c800a99d9d4bf0f
Nov 03 18:36:02.822538 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.822500    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7" event={"ID":"073e1231-6e9f-481b-9f9d-fbd01d8b2c33","Type":"ContainerStarted","Data":"17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411"}
Nov 03 18:36:02.822694 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.822548    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7" event={"ID":"073e1231-6e9f-481b-9f9d-fbd01d8b2c33","Type":"ContainerStarted","Data":"cab90450dd22c788300244da2366ef55d090184ac75ee0d54c800a99d9d4bf0f"}
Nov 03 18:36:02.945784 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:02.945726    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7" podStartSLOduration=1.945712495 podStartE2EDuration="1.945712495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:02.944493314 +0000 UTC m=+3742.403664307" watchObservedRunningTime="2026-11-03 18:36:02.945712495 +0000 UTC m=+3742.404883493"
Nov 03 18:36:03.356921 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:03.356890    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:06.994662 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:36:06.994622    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:36:12.357665 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:12.357627    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:12.370758 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:12.370725    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:12.875277 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:12.875253    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:36:36.991454 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:36:36.991418    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:36:43.744613 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:36:43.744583    4493 scope.go:122] "RemoveContainer" containerID="5b3f12ab3117046333c15037ed0311243073dd2a83e9809b78e56aa896ab9730"
Nov 03 18:37:02.053711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:02.053678    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"]
Nov 03 18:37:06.991467 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:37:06.991430    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:37:17.073650 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.073587    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7" podUID="073e1231-6e9f-481b-9f9d-fbd01d8b2c33" containerName="main" containerID="cri-o://17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411" gracePeriod=45
Nov 03 18:37:17.315767 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.315740    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:37:17.395490 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395410    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tls-certs\" (UniqueName: \"kubernetes.io/secret/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tls-certs\") pod \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") "
Nov 03 18:37:17.395490 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395460    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-model-cache\") pod \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") "
Nov 03 18:37:17.395688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395522    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-dshm\") pod \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") "
Nov 03 18:37:17.395688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395554    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-kube-api-access-qq6z6\" (UniqueName: \"kubernetes.io/projected/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-kube-api-access-qq6z6\") pod \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") "
Nov 03 18:37:17.395688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395635    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-home\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-home\") pod \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") "
Nov 03 18:37:17.395688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395675    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tmp-dir\") pod \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\" (UID: \"073e1231-6e9f-481b-9f9d-fbd01d8b2c33\") "
Nov 03 18:37:17.395949 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395740    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-model-cache" pod "073e1231-6e9f-481b-9f9d-fbd01d8b2c33" (UID: "073e1231-6e9f-481b-9f9d-fbd01d8b2c33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:17.396044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.395980    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-home" pod "073e1231-6e9f-481b-9f9d-fbd01d8b2c33" (UID: "073e1231-6e9f-481b-9f9d-fbd01d8b2c33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:17.396044 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.396008    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:17.396462 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.396074    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tmp-dir" pod "073e1231-6e9f-481b-9f9d-fbd01d8b2c33" (UID: "073e1231-6e9f-481b-9f9d-fbd01d8b2c33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:17.397862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.397842    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-dshm" pod "073e1231-6e9f-481b-9f9d-fbd01d8b2c33" (UID: "073e1231-6e9f-481b-9f9d-fbd01d8b2c33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:37:17.398024 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.398003    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tls-certs" pod "073e1231-6e9f-481b-9f9d-fbd01d8b2c33" (UID: "073e1231-6e9f-481b-9f9d-fbd01d8b2c33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:37:17.398140 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.398114    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-kube-api-access-qq6z6" pod "073e1231-6e9f-481b-9f9d-fbd01d8b2c33" (UID: "073e1231-6e9f-481b-9f9d-fbd01d8b2c33"). InnerVolumeSpecName "kube-api-access-qq6z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:37:17.497562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.497526    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:17.497562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.497555    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:17.497562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.497566    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:17.497562 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.497573    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:17.497961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:17.497586    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq6z6\" (UniqueName: \"kubernetes.io/projected/073e1231-6e9f-481b-9f9d-fbd01d8b2c33-kube-api-access-qq6z6\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:37:18.137489 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.137454    4493 generic.go:356] "Generic (PLEG): container finished" podID="073e1231-6e9f-481b-9f9d-fbd01d8b2c33" containerID="17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411" exitCode=0
Nov 03 18:37:18.137942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.137499    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"
Nov 03 18:37:18.137942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.137517    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7" event={"ID":"073e1231-6e9f-481b-9f9d-fbd01d8b2c33","Type":"ContainerDied","Data":"17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411"}
Nov 03 18:37:18.137942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.137550    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7" event={"ID":"073e1231-6e9f-481b-9f9d-fbd01d8b2c33","Type":"ContainerDied","Data":"cab90450dd22c788300244da2366ef55d090184ac75ee0d54c800a99d9d4bf0f"}
Nov 03 18:37:18.137942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.137567    4493 scope.go:122] "RemoveContainer" containerID="17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411"
Nov 03 18:37:18.146340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.146324    4493 scope.go:122] "RemoveContainer" containerID="17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411"
Nov 03 18:37:18.146540 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:37:18.146521    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411\": container with ID starting with 17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411 not found: ID does not exist" containerID="17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411"
Nov 03 18:37:18.146608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.146548    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411"} err="failed to get container status \"17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411\": rpc error: code = NotFound desc = could not find container \"17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411\": container with ID starting with 17589af442435308f2302825e4483871185daab8b4f5b6b6b2c01d1c47f11411 not found: ID does not exist"
Nov 03 18:37:18.306296 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.306265    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"]
Nov 03 18:37:18.356709 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.356674    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-057118/simulator-single-main-057118-kserve-57d5f74c76-vngk7"]
Nov 03 18:37:18.710961 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:18.710930    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073e1231-6e9f-481b-9f9d-fbd01d8b2c33" path="/var/lib/kubelet/pods/073e1231-6e9f-481b-9f9d-fbd01d8b2c33/volumes"
Nov 03 18:37:29.604061 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.603967    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"]
Nov 03 18:37:29.604540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.604459    4493 cpu_manager.go:414] "containerMap: removing container" podUID="073e1231-6e9f-481b-9f9d-fbd01d8b2c33" containerName="main"
Nov 03 18:37:29.604540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.604477    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073e1231-6e9f-481b-9f9d-fbd01d8b2c33" containerName="main"
Nov 03 18:37:29.604663 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.604582    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="073e1231-6e9f-481b-9f9d-fbd01d8b2c33" containerName="main"
Nov 03 18:37:29.609037 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.609013    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.664093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.664060    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 18:37:29.710729 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.710683    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6939bf30-61a0-4d3a-a39f-260a94349897-tls-certs\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.710948 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.710750    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqxhv\" (UniqueName: \"kubernetes.io/projected/6939bf30-61a0-4d3a-a39f-260a94349897-kube-api-access-rqxhv\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.710948 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.710843    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-dshm\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.710948 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.710902    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-tmp-dir\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.710948 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.710922    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-model-cache\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.711117 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.711032    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-home\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.713861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.713838    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 18:37:29.714031 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.713896    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-52z8v\""
Nov 03 18:37:29.715358 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.715332    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"]
Nov 03 18:37:29.764804 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.764774    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a9a93c65-kserve-self-signed-certs\""
Nov 03 18:37:29.811538 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.811499    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6939bf30-61a0-4d3a-a39f-260a94349897-tls-certs\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.811711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.811563    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqxhv\" (UniqueName: \"kubernetes.io/projected/6939bf30-61a0-4d3a-a39f-260a94349897-kube-api-access-rqxhv\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.811711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.811589    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-dshm\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.811711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.811609    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-tmp-dir\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.811711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.811625    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-model-cache\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.811711 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.811668    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-home\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.812218 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.812189    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-tmp-dir\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.812357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.812227    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-home\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.812357 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.812268    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-model-cache\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.814146 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.814119    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-dshm\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.814560 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.814539    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6939bf30-61a0-4d3a-a39f-260a94349897-tls-certs\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.877497 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.877420    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqxhv\" (UniqueName: \"kubernetes.io/projected/6939bf30-61a0-4d3a-a39f-260a94349897-kube-api-access-rqxhv\") pod \"e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") " pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:29.920544 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:29.920503    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:30.300412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:30.300241    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"]
Nov 03 18:37:30.303052 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:37:30.303022    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6939bf30_61a0_4d3a_a39f_260a94349897.slice/crio-d50869983c4e622be160231550ab7fd6fa0d8d2556b3f33702a0702d2266ddaa WatchSource:0}: Error finding container d50869983c4e622be160231550ab7fd6fa0d8d2556b3f33702a0702d2266ddaa: Status 404 returned error can't find the container with id d50869983c4e622be160231550ab7fd6fa0d8d2556b3f33702a0702d2266ddaa
Nov 03 18:37:31.193316 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:31.193276    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps" event={"ID":"6939bf30-61a0-4d3a-a39f-260a94349897","Type":"ContainerStarted","Data":"340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc"}
Nov 03 18:37:31.193737 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:31.193326    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps" event={"ID":"6939bf30-61a0-4d3a-a39f-260a94349897","Type":"ContainerStarted","Data":"d50869983c4e622be160231550ab7fd6fa0d8d2556b3f33702a0702d2266ddaa"}
Nov 03 18:37:31.315990 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:31.315927    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps" podStartSLOduration=2.315910613 podStartE2EDuration="2.315910613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:31.314622381 +0000 UTC m=+3830.773793405" watchObservedRunningTime="2026-11-03 18:37:31.315910613 +0000 UTC m=+3830.775081599"
Nov 03 18:37:31.921076 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:31.921034    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:37.008812 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:37:37.008771    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:37:39.921064 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:39.921028    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:39.934000 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:39.933970    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:37:40.253552 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:37:40.253520    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:38:06.993022 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:38:06.992976    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:38:15.214397 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.214364    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"]
Nov 03 18:38:15.217752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.217736    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.261622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.261584    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-tmp-dir\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.261814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.261639    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda434c7-d222-4a15-a315-433d8fcb20c5-tls-certs\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.261814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.261700    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72pn\" (UniqueName: \"kubernetes.io/projected/fda434c7-d222-4a15-a315-433d8fcb20c5-kube-api-access-s72pn\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.261814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.261734    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-model-cache\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.261814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.261765    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-home\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.262005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.261818    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-dshm\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.322660 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.322612    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a9a93c65-kserve-self-signed-certs\""
Nov 03 18:38:15.325786 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.325752    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"]
Nov 03 18:38:15.363103 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363064    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-tmp-dir\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363303 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363148    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda434c7-d222-4a15-a315-433d8fcb20c5-tls-certs\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363303 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363187    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72pn\" (UniqueName: \"kubernetes.io/projected/fda434c7-d222-4a15-a315-433d8fcb20c5-kube-api-access-s72pn\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363303 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363239    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-model-cache\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363303 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363285    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-home\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363318    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-dshm\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363543 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363522    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-tmp-dir\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363723    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-model-cache\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.363847 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.363827    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-home\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.365901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.365860    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda434c7-d222-4a15-a315-433d8fcb20c5-tls-certs\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.366010 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.365919    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-dshm\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.427578 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.427542    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72pn\" (UniqueName: \"kubernetes.io/projected/fda434c7-d222-4a15-a315-433d8fcb20c5-kube-api-access-s72pn\") pod \"e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") " pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.529315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.529217    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:15.916985 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:15.916959    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"]
Nov 03 18:38:15.919439 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:38:15.919403    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfda434c7_d222_4a15_a315_433d8fcb20c5.slice/crio-6f11cd57ef3e2c39e265e8c3414892a086c7b05f2c4974e6b68ec3cb168b8bdf WatchSource:0}: Error finding container 6f11cd57ef3e2c39e265e8c3414892a086c7b05f2c4974e6b68ec3cb168b8bdf: Status 404 returned error can't find the container with id 6f11cd57ef3e2c39e265e8c3414892a086c7b05f2c4974e6b68ec3cb168b8bdf
Nov 03 18:38:16.389642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:16.389603    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg" event={"ID":"fda434c7-d222-4a15-a315-433d8fcb20c5","Type":"ContainerStarted","Data":"366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5"}
Nov 03 18:38:16.389642 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:16.389648    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg" event={"ID":"fda434c7-d222-4a15-a315-433d8fcb20c5","Type":"ContainerStarted","Data":"6f11cd57ef3e2c39e265e8c3414892a086c7b05f2c4974e6b68ec3cb168b8bdf"}
Nov 03 18:38:16.511816 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:16.511764    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg" podStartSLOduration=1.511748435 podStartE2EDuration="1.511748435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:16.509530906 +0000 UTC m=+3875.968701906" watchObservedRunningTime="2026-11-03 18:38:16.511748435 +0000 UTC m=+3875.970919399"
Nov 03 18:38:16.529931 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:16.529896    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:25.530194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:25.530153    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:25.542609 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:25.542579    4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:26.464605 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:26.464575    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:38:36.992630 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:38:36.992588    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:38:59.021386 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:38:59.021303    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"]
Nov 03 18:39:06.994788 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:39:06.994740    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:39:14.041438 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.041366    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps" podUID="6939bf30-61a0-4d3a-a39f-260a94349897" containerName="main" containerID="cri-o://340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc" gracePeriod=45
Nov 03 18:39:14.281304 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.281275    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:39:14.442757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.442664    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6939bf30-61a0-4d3a-a39f-260a94349897-tls-certs\" (UniqueName: \"kubernetes.io/secret/6939bf30-61a0-4d3a-a39f-260a94349897-tls-certs\") pod \"6939bf30-61a0-4d3a-a39f-260a94349897\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") "
Nov 03 18:39:14.442757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.442710    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-tmp-dir\") pod \"6939bf30-61a0-4d3a-a39f-260a94349897\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") "
Nov 03 18:39:14.442757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.442743    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-dshm\") pod \"6939bf30-61a0-4d3a-a39f-260a94349897\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") "
Nov 03 18:39:14.443104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.442821    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-model-cache\") pod \"6939bf30-61a0-4d3a-a39f-260a94349897\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") "
Nov 03 18:39:14.443104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.442853    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-home\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-home\") pod \"6939bf30-61a0-4d3a-a39f-260a94349897\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") "
Nov 03 18:39:14.443104 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.442942    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6939bf30-61a0-4d3a-a39f-260a94349897-kube-api-access-rqxhv\" (UniqueName: \"kubernetes.io/projected/6939bf30-61a0-4d3a-a39f-260a94349897-kube-api-access-rqxhv\") pod \"6939bf30-61a0-4d3a-a39f-260a94349897\" (UID: \"6939bf30-61a0-4d3a-a39f-260a94349897\") "
Nov 03 18:39:14.443269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.443139    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-tmp-dir" pod "6939bf30-61a0-4d3a-a39f-260a94349897" (UID: "6939bf30-61a0-4d3a-a39f-260a94349897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:14.443269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.443153    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-model-cache" pod "6939bf30-61a0-4d3a-a39f-260a94349897" (UID: "6939bf30-61a0-4d3a-a39f-260a94349897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:14.443269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.443170    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-home" pod "6939bf30-61a0-4d3a-a39f-260a94349897" (UID: "6939bf30-61a0-4d3a-a39f-260a94349897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:14.445233 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.445201    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6939bf30-61a0-4d3a-a39f-260a94349897-kube-api-access-rqxhv" pod "6939bf30-61a0-4d3a-a39f-260a94349897" (UID: "6939bf30-61a0-4d3a-a39f-260a94349897"). InnerVolumeSpecName "kube-api-access-rqxhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:14.445338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.445229    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6939bf30-61a0-4d3a-a39f-260a94349897-tls-certs" pod "6939bf30-61a0-4d3a-a39f-260a94349897" (UID: "6939bf30-61a0-4d3a-a39f-260a94349897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:39:14.445338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.445288    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-dshm" pod "6939bf30-61a0-4d3a-a39f-260a94349897" (UID: "6939bf30-61a0-4d3a-a39f-260a94349897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:14.544386 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.544341    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:14.544386 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.544377    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:14.544602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.544392    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqxhv\" (UniqueName: \"kubernetes.io/projected/6939bf30-61a0-4d3a-a39f-260a94349897-kube-api-access-rqxhv\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:14.544602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.544405    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6939bf30-61a0-4d3a-a39f-260a94349897-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:14.544602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.544416    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:14.544602 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.544428    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6939bf30-61a0-4d3a-a39f-260a94349897-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:14.688658 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.688620    4493 generic.go:356] "Generic (PLEG): container finished" podID="6939bf30-61a0-4d3a-a39f-260a94349897" containerID="340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc" exitCode=0
Nov 03 18:39:14.688861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.688669    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"
Nov 03 18:39:14.688861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.688707    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps" event={"ID":"6939bf30-61a0-4d3a-a39f-260a94349897","Type":"ContainerDied","Data":"340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc"}
Nov 03 18:39:14.688861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.688754    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps" event={"ID":"6939bf30-61a0-4d3a-a39f-260a94349897","Type":"ContainerDied","Data":"d50869983c4e622be160231550ab7fd6fa0d8d2556b3f33702a0702d2266ddaa"}
Nov 03 18:39:14.688861 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.688771    4493 scope.go:122] "RemoveContainer" containerID="340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc"
Nov 03 18:39:14.697418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.697401    4493 scope.go:122] "RemoveContainer" containerID="340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc"
Nov 03 18:39:14.697709 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:39:14.697688    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc\": container with ID starting with 340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc not found: ID does not exist" containerID="340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc"
Nov 03 18:39:14.697766 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.697720    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc"} err="failed to get container status \"340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc\": rpc error: code = NotFound desc = could not find container \"340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc\": container with ID starting with 340ff11b6a619a0984cec4820dfbed3de359f8b72d9b32a598907618bef1dedc not found: ID does not exist"
Nov 03 18:39:14.857769 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.857733    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"]
Nov 03 18:39:14.908418 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:14.908385    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps"]
Nov 03 18:39:15.242678 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:15.242631    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-a9a93c65-kserve-74b865cb85-qn9ps" podUID="6939bf30-61a0-4d3a-a39f-260a94349897" containerName="main" probeResult="failure" output="Get \"https://10.133.0.219:8000/ready\": context deadline exceeded"
Nov 03 18:39:16.710197 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:16.710164    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6939bf30-61a0-4d3a-a39f-260a94349897" path="/var/lib/kubelet/pods/6939bf30-61a0-4d3a-a39f-260a94349897/volumes"
Nov 03 18:39:17.812942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:17.812907    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"]
Nov 03 18:39:32.832690 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:32.832622    4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg" podUID="fda434c7-d222-4a15-a315-433d8fcb20c5" containerName="main" containerID="cri-o://366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5" gracePeriod=45
Nov 03 18:39:33.090254 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.090187    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:39:33.158304 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158268    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-home\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-home\") pod \"fda434c7-d222-4a15-a315-433d8fcb20c5\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") "
Nov 03 18:39:33.158494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158364    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda434c7-d222-4a15-a315-433d8fcb20c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/fda434c7-d222-4a15-a315-433d8fcb20c5-tls-certs\") pod \"fda434c7-d222-4a15-a315-433d8fcb20c5\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") "
Nov 03 18:39:33.158494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158417    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda434c7-d222-4a15-a315-433d8fcb20c5-kube-api-access-s72pn\" (UniqueName: \"kubernetes.io/projected/fda434c7-d222-4a15-a315-433d8fcb20c5-kube-api-access-s72pn\") pod \"fda434c7-d222-4a15-a315-433d8fcb20c5\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") "
Nov 03 18:39:33.158494 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158439    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-model-cache\") pod \"fda434c7-d222-4a15-a315-433d8fcb20c5\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") "
Nov 03 18:39:33.158671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158518    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-tmp-dir\") pod \"fda434c7-d222-4a15-a315-433d8fcb20c5\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") "
Nov 03 18:39:33.158671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158580    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-dshm\") pod \"fda434c7-d222-4a15-a315-433d8fcb20c5\" (UID: \"fda434c7-d222-4a15-a315-433d8fcb20c5\") "
Nov 03 18:39:33.158671 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158594    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-home" pod "fda434c7-d222-4a15-a315-433d8fcb20c5" (UID: "fda434c7-d222-4a15-a315-433d8fcb20c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:33.158819 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158677    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-model-cache" pod "fda434c7-d222-4a15-a315-433d8fcb20c5" (UID: "fda434c7-d222-4a15-a315-433d8fcb20c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:33.158819 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158784    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-tmp-dir" pod "fda434c7-d222-4a15-a315-433d8fcb20c5" (UID: "fda434c7-d222-4a15-a315-433d8fcb20c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:33.158983 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158962    4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-home\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:33.159050 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.158988    4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-model-cache\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:33.159050 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.159001    4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-tmp-dir\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:33.160749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.160722    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda434c7-d222-4a15-a315-433d8fcb20c5-tls-certs" pod "fda434c7-d222-4a15-a315-433d8fcb20c5" (UID: "fda434c7-d222-4a15-a315-433d8fcb20c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:39:33.160749 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.160734    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda434c7-d222-4a15-a315-433d8fcb20c5-kube-api-access-s72pn" pod "fda434c7-d222-4a15-a315-433d8fcb20c5" (UID: "fda434c7-d222-4a15-a315-433d8fcb20c5"). InnerVolumeSpecName "kube-api-access-s72pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:39:33.160907 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.160783    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-dshm" pod "fda434c7-d222-4a15-a315-433d8fcb20c5" (UID: "fda434c7-d222-4a15-a315-433d8fcb20c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:39:33.259795 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.259753    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s72pn\" (UniqueName: \"kubernetes.io/projected/fda434c7-d222-4a15-a315-433d8fcb20c5-kube-api-access-s72pn\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:33.259795 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.259785    4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda434c7-d222-4a15-a315-433d8fcb20c5-dshm\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:33.259795 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.259797    4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda434c7-d222-4a15-a315-433d8fcb20c5-tls-certs\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:39:33.805301 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.805266    4493 generic.go:356] "Generic (PLEG): container finished" podID="fda434c7-d222-4a15-a315-433d8fcb20c5" containerID="366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5" exitCode=0
Nov 03 18:39:33.805473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.805322    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"
Nov 03 18:39:33.805473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.805355    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg" event={"ID":"fda434c7-d222-4a15-a315-433d8fcb20c5","Type":"ContainerDied","Data":"366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5"}
Nov 03 18:39:33.805473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.805406    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg" event={"ID":"fda434c7-d222-4a15-a315-433d8fcb20c5","Type":"ContainerDied","Data":"6f11cd57ef3e2c39e265e8c3414892a086c7b05f2c4974e6b68ec3cb168b8bdf"}
Nov 03 18:39:33.805473 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.805424    4493 scope.go:122] "RemoveContainer" containerID="366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5"
Nov 03 18:39:33.814194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.814175    4493 scope.go:122] "RemoveContainer" containerID="366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5"
Nov 03 18:39:33.814488 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:39:33.814467    4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5\": container with ID starting with 366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5 not found: ID does not exist" containerID="366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5"
Nov 03 18:39:33.814558 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.814502    4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5"} err="failed to get container status \"366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5\": rpc error: code = NotFound desc = could not find container \"366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5\": container with ID starting with 366a3c39b5a1c45018732112857f31448f839a2145eb3f82ab18c28cc5ce3fc5 not found: ID does not exist"
Nov 03 18:39:33.975135 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.975105    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"]
Nov 03 18:39:33.981571 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:33.981547    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a9a93c65-kserve-67b5949f5c-nvgvg"]
Nov 03 18:39:34.711269 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:39:34.711236    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda434c7-d222-4a15-a315-433d8fcb20c5" path="/var/lib/kubelet/pods/fda434c7-d222-4a15-a315-433d8fcb20c5/volumes"
Nov 03 18:39:36.994049 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:39:36.994008    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:40:06.991398 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:40:06.991354    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:40:36.991532 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:40:36.991492    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:40:55.197736 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.197699    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"]
Nov 03 18:40:55.198236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.198046    4493 cpu_manager.go:414] "containerMap: removing container" podUID="6939bf30-61a0-4d3a-a39f-260a94349897" containerName="main"
Nov 03 18:40:55.198236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.198057    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6939bf30-61a0-4d3a-a39f-260a94349897" containerName="main"
Nov 03 18:40:55.198236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.198068    4493 cpu_manager.go:414] "containerMap: removing container" podUID="fda434c7-d222-4a15-a315-433d8fcb20c5" containerName="main"
Nov 03 18:40:55.198236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.198073    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda434c7-d222-4a15-a315-433d8fcb20c5" containerName="main"
Nov 03 18:40:55.198236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.198132    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6939bf30-61a0-4d3a-a39f-260a94349897" containerName="main"
Nov 03 18:40:55.198236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.198140    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda434c7-d222-4a15-a315-433d8fcb20c5" containerName="main"
Nov 03 18:40:55.200597 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.200580    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.256653 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.256611    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:40:55.306349 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.306313    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7s4vb\""
Nov 03 18:40:55.306349 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.306339    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:40:55.306557 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.306313    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:40:55.309580 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.309558    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"]
Nov 03 18:40:55.362027 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.361991    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84bc\" (UniqueName: \"kubernetes.io/projected/344f6176-6dd5-4fc9-a6ef-c9587acb7216-kube-api-access-h84bc\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.362212 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.362057    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.362264 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.362229    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.463088 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.463058    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.463285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.463100    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84bc\" (UniqueName: \"kubernetes.io/projected/344f6176-6dd5-4fc9-a6ef-c9587acb7216-kube-api-access-h84bc\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.463285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.463140    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.463724 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.463703    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.465653 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.465628    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.527391 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.527361    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84bc\" (UniqueName: \"kubernetes.io/projected/344f6176-6dd5-4fc9-a6ef-c9587acb7216-kube-api-access-h84bc\") pod \"maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:55.813071 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:55.812994    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:56.184908 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:56.184858    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"]
Nov 03 18:40:56.187539 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:40:56.187513    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344f6176_6dd5_4fc9_a6ef_c9587acb7216.slice/crio-7198550186be557e1b4770f52012a0b36319e8483f7494363536a1535ceecdec WatchSource:0}: Error finding container 7198550186be557e1b4770f52012a0b36319e8483f7494363536a1535ceecdec: Status 404 returned error can't find the container with id 7198550186be557e1b4770f52012a0b36319e8483f7494363536a1535ceecdec
Nov 03 18:40:57.147540 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:57.147510    4493 generic.go:356] "Generic (PLEG): container finished" podID="344f6176-6dd5-4fc9-a6ef-c9587acb7216" containerID="4e1714c28187ebbb6e7e6b4cfd978739398e179426de7203bd74409af51bb272" exitCode=0
Nov 03 18:40:57.148011 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:57.147599    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9" event={"ID":"344f6176-6dd5-4fc9-a6ef-c9587acb7216","Type":"ContainerDied","Data":"4e1714c28187ebbb6e7e6b4cfd978739398e179426de7203bd74409af51bb272"}
Nov 03 18:40:57.148011 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:57.147648    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9" event={"ID":"344f6176-6dd5-4fc9-a6ef-c9587acb7216","Type":"ContainerStarted","Data":"7198550186be557e1b4770f52012a0b36319e8483f7494363536a1535ceecdec"}
Nov 03 18:40:58.283536 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.283508    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:58.388238 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.388202    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-xks-ca\") pod \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") "
Nov 03 18:40:58.388238 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.388241    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-openshift-ca\") pod \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") "
Nov 03 18:40:58.388495 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.388329    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344f6176-6dd5-4fc9-a6ef-c9587acb7216-kube-api-access-h84bc\" (UniqueName: \"kubernetes.io/projected/344f6176-6dd5-4fc9-a6ef-c9587acb7216-kube-api-access-h84bc\") pod \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\" (UID: \"344f6176-6dd5-4fc9-a6ef-c9587acb7216\") "
Nov 03 18:40:58.388688 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.388662    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-openshift-ca" pod "344f6176-6dd5-4fc9-a6ef-c9587acb7216" (UID: "344f6176-6dd5-4fc9-a6ef-c9587acb7216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:40:58.390441 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.390409    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-xks-ca" pod "344f6176-6dd5-4fc9-a6ef-c9587acb7216" (UID: "344f6176-6dd5-4fc9-a6ef-c9587acb7216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:40:58.390534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.390489    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344f6176-6dd5-4fc9-a6ef-c9587acb7216-kube-api-access-h84bc" pod "344f6176-6dd5-4fc9-a6ef-c9587acb7216" (UID: "344f6176-6dd5-4fc9-a6ef-c9587acb7216"). InnerVolumeSpecName "kube-api-access-h84bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:40:58.489219 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.489188    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84bc\" (UniqueName: \"kubernetes.io/projected/344f6176-6dd5-4fc9-a6ef-c9587acb7216-kube-api-access-h84bc\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:40:58.489219 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.489219    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:40:58.489419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:58.489228    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/344f6176-6dd5-4fc9-a6ef-c9587acb7216-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:40:59.155849 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:59.155815    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9" event={"ID":"344f6176-6dd5-4fc9-a6ef-c9587acb7216","Type":"ContainerDied","Data":"7198550186be557e1b4770f52012a0b36319e8483f7494363536a1535ceecdec"}
Nov 03 18:40:59.156051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:59.155854    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"
Nov 03 18:40:59.156051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:40:59.155862    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7198550186be557e1b4770f52012a0b36319e8483f7494363536a1535ceecdec"
Nov 03 18:41:06.991822 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:41:06.991785    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:41:17.122935 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:41:17.122897    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"]
Nov 03 18:41:17.173353 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:41:17.173309    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c188b2e5-sq8v9"]
Nov 03 18:41:18.710289 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:41:18.710257    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344f6176-6dd5-4fc9-a6ef-c9587acb7216" path="/var/lib/kubelet/pods/344f6176-6dd5-4fc9-a6ef-c9587acb7216/volumes"
Nov 03 18:41:36.991510 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:41:36.991461    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:42:06.992114 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:42:06.992024    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:42:36.992043 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:42:36.992003    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:43:06.991312 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:43:06.991277    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:43:12.899144 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:12.899112    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"]
Nov 03 18:43:12.899534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:12.899475    4493 cpu_manager.go:414] "containerMap: removing container" podUID="344f6176-6dd5-4fc9-a6ef-c9587acb7216" containerName="revoke-keys"
Nov 03 18:43:12.899534 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:12.899494    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344f6176-6dd5-4fc9-a6ef-c9587acb7216" containerName="revoke-keys"
Nov 03 18:43:12.899608 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:12.899580    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="344f6176-6dd5-4fc9-a6ef-c9587acb7216" containerName="revoke-keys"
Nov 03 18:43:12.902610 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:12.902585    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:12.957620 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:12.957589    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:43:13.004236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.004207    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmch8\" (UniqueName: \"kubernetes.io/projected/4ff06906-97a4-4292-abae-c8cdf0b85dbc-kube-api-access-dmch8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.004390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.004278    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.004390 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.004341    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.006851 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.006833    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:43:13.006966 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.006924    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:43:13.007009 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.006995    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7s4vb\""
Nov 03 18:43:13.008501 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.008484    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"]
Nov 03 18:43:13.105179 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.105153    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmch8\" (UniqueName: \"kubernetes.io/projected/4ff06906-97a4-4292-abae-c8cdf0b85dbc-kube-api-access-dmch8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.105338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.105216    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.105338 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.105237    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.105908 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.105861    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.107707 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.107678    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.168452 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.168387    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmch8\" (UniqueName: \"kubernetes.io/projected/4ff06906-97a4-4292-abae-c8cdf0b85dbc-kube-api-access-dmch8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.215008 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.214984    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:13.586284 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.586260    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"]
Nov 03 18:43:13.588847 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:43:13.588817    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff06906_97a4_4292_abae_c8cdf0b85dbc.slice/crio-ecd7d639190601ce5c23aca81e69085c0693d2532f468b6bff14d14beff0106d WatchSource:0}: Error finding container ecd7d639190601ce5c23aca81e69085c0693d2532f468b6bff14d14beff0106d: Status 404 returned error can't find the container with id ecd7d639190601ce5c23aca81e69085c0693d2532f468b6bff14d14beff0106d
Nov 03 18:43:13.708662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:13.708626    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh" event={"ID":"4ff06906-97a4-4292-abae-c8cdf0b85dbc","Type":"ContainerStarted","Data":"ecd7d639190601ce5c23aca81e69085c0693d2532f468b6bff14d14beff0106d"}
Nov 03 18:43:14.713652 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:14.713620    4493 generic.go:356] "Generic (PLEG): container finished" podID="4ff06906-97a4-4292-abae-c8cdf0b85dbc" containerID="e35e4e38def31ac75c18b24bac33b4dbed609c3fbd4726b135d987e35e4d7299" exitCode=0
Nov 03 18:43:14.714067 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:14.713657    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh" event={"ID":"4ff06906-97a4-4292-abae-c8cdf0b85dbc","Type":"ContainerDied","Data":"e35e4e38def31ac75c18b24bac33b4dbed609c3fbd4726b135d987e35e4d7299"}
Nov 03 18:43:15.844564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:15.844534    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:15.924716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:15.924681    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ff06906-97a4-4292-abae-c8cdf0b85dbc-kube-api-access-dmch8\" (UniqueName: \"kubernetes.io/projected/4ff06906-97a4-4292-abae-c8cdf0b85dbc-kube-api-access-dmch8\") pod \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") "
Nov 03 18:43:15.924901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:15.924723    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-xks-ca\") pod \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") "
Nov 03 18:43:15.924901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:15.924773    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-openshift-ca\") pod \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\" (UID: \"4ff06906-97a4-4292-abae-c8cdf0b85dbc\") "
Nov 03 18:43:15.925178 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:15.925154    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-openshift-ca" pod "4ff06906-97a4-4292-abae-c8cdf0b85dbc" (UID: "4ff06906-97a4-4292-abae-c8cdf0b85dbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:43:15.927026 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:15.927000    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-xks-ca" pod "4ff06906-97a4-4292-abae-c8cdf0b85dbc" (UID: "4ff06906-97a4-4292-abae-c8cdf0b85dbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:43:15.927128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:15.927052    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff06906-97a4-4292-abae-c8cdf0b85dbc-kube-api-access-dmch8" pod "4ff06906-97a4-4292-abae-c8cdf0b85dbc" (UID: "4ff06906-97a4-4292-abae-c8cdf0b85dbc"). InnerVolumeSpecName "kube-api-access-dmch8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:43:16.025564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:16.025488    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmch8\" (UniqueName: \"kubernetes.io/projected/4ff06906-97a4-4292-abae-c8cdf0b85dbc-kube-api-access-dmch8\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:16.025564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:16.025518    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:16.025564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:16.025528    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ff06906-97a4-4292-abae-c8cdf0b85dbc-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:43:16.722051 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:16.722026    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"
Nov 03 18:43:16.722213 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:16.722029    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh" event={"ID":"4ff06906-97a4-4292-abae-c8cdf0b85dbc","Type":"ContainerDied","Data":"ecd7d639190601ce5c23aca81e69085c0693d2532f468b6bff14d14beff0106d"}
Nov 03 18:43:16.722213 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:16.722149    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecd7d639190601ce5c23aca81e69085c0693d2532f468b6bff14d14beff0106d"
Nov 03 18:43:36.992145 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:43:36.992103    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:43:37.108783 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:37.108745    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"]
Nov 03 18:43:37.159557 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:37.159523    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8d4263-fxjdh"]
Nov 03 18:43:38.710996 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:43:38.710961    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ff06906-97a4-4292-abae-c8cdf0b85dbc" path="/var/lib/kubelet/pods/4ff06906-97a4-4292-abae-c8cdf0b85dbc/volumes"
Nov 03 18:44:07.009218 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:44:07.009179    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:44:12.943401 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:12.943361    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"]
Nov 03 18:44:12.943909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:12.943699    4493 cpu_manager.go:414] "containerMap: removing container" podUID="4ff06906-97a4-4292-abae-c8cdf0b85dbc" containerName="revoke-keys"
Nov 03 18:44:12.943909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:12.943710    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ff06906-97a4-4292-abae-c8cdf0b85dbc" containerName="revoke-keys"
Nov 03 18:44:12.943909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:12.943778    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ff06906-97a4-4292-abae-c8cdf0b85dbc" containerName="revoke-keys"
Nov 03 18:44:12.946564 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:12.946548    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.001462 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.001431    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:44:13.051554 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.051521    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:44:13.051554 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.051520    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:44:13.051778 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.051594    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7s4vb\""
Nov 03 18:44:13.053025 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.053006    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"]
Nov 03 18:44:13.064410 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.064387    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.064555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.064422    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbn7\" (UniqueName: \"kubernetes.io/projected/b00e03bc-6936-4d42-bf9b-60741aa668ba-kube-api-access-5bbn7\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.064555 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.064458    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.165925 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.165863    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.166093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.165942    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbn7\" (UniqueName: \"kubernetes.io/projected/b00e03bc-6936-4d42-bf9b-60741aa668ba-kube-api-access-5bbn7\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.166093 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.165980    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.166638 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.166615    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.168539 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.168511    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.229767 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.229730    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbn7\" (UniqueName: \"kubernetes.io/projected/b00e03bc-6936-4d42-bf9b-60741aa668ba-kube-api-access-5bbn7\") pod \"maas-api-revoke-sub-f03629db-gwsj2\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.258929 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.258899    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:13.633888 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.633773    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"]
Nov 03 18:44:13.636584 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:44:13.636555    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00e03bc_6936_4d42_bf9b_60741aa668ba.slice/crio-ad5588f6711bd3c65b8a3934f98713631f1303653e484944d7f67076bd28c0d0 WatchSource:0}: Error finding container ad5588f6711bd3c65b8a3934f98713631f1303653e484944d7f67076bd28c0d0: Status 404 returned error can't find the container with id ad5588f6711bd3c65b8a3934f98713631f1303653e484944d7f67076bd28c0d0
Nov 03 18:44:13.991372 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.991331    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2" event={"ID":"b00e03bc-6936-4d42-bf9b-60741aa668ba","Type":"ContainerStarted","Data":"2b895b35e5f93652249f48b2748d50bc0acfe32de08209116d3fe9f3605a6241"}
Nov 03 18:44:13.991808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:13.991397    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2" event={"ID":"b00e03bc-6936-4d42-bf9b-60741aa668ba","Type":"ContainerStarted","Data":"ad5588f6711bd3c65b8a3934f98713631f1303653e484944d7f67076bd28c0d0"}
Nov 03 18:44:14.111933 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:14.111856    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2" podStartSLOduration=2.111842682 podStartE2EDuration="2.111842682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:14.111099646 +0000 UTC m=+4233.570270632" watchObservedRunningTime="2026-11-03 18:44:14.111842682 +0000 UTC m=+4233.571013677"
Nov 03 18:44:14.996354 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:14.996317    4493 generic.go:356] "Generic (PLEG): container finished" podID="b00e03bc-6936-4d42-bf9b-60741aa668ba" containerID="2b895b35e5f93652249f48b2748d50bc0acfe32de08209116d3fe9f3605a6241" exitCode=0
Nov 03 18:44:14.996778 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:14.996366    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2" event={"ID":"b00e03bc-6936-4d42-bf9b-60741aa668ba","Type":"ContainerDied","Data":"2b895b35e5f93652249f48b2748d50bc0acfe32de08209116d3fe9f3605a6241"}
Nov 03 18:44:16.130274 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.130250    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:16.193648 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.193615    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00e03bc-6936-4d42-bf9b-60741aa668ba-kube-api-access-5bbn7\" (UniqueName: \"kubernetes.io/projected/b00e03bc-6936-4d42-bf9b-60741aa668ba-kube-api-access-5bbn7\") pod \"b00e03bc-6936-4d42-bf9b-60741aa668ba\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") "
Nov 03 18:44:16.193777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.193662    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-openshift-ca\") pod \"b00e03bc-6936-4d42-bf9b-60741aa668ba\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") "
Nov 03 18:44:16.193777 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.193692    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-xks-ca\") pod \"b00e03bc-6936-4d42-bf9b-60741aa668ba\" (UID: \"b00e03bc-6936-4d42-bf9b-60741aa668ba\") "
Nov 03 18:44:16.194131 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.194106    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-openshift-ca" pod "b00e03bc-6936-4d42-bf9b-60741aa668ba" (UID: "b00e03bc-6936-4d42-bf9b-60741aa668ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:44:16.196015 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.195991    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-xks-ca" pod "b00e03bc-6936-4d42-bf9b-60741aa668ba" (UID: "b00e03bc-6936-4d42-bf9b-60741aa668ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:44:16.196134 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.196065    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00e03bc-6936-4d42-bf9b-60741aa668ba-kube-api-access-5bbn7" pod "b00e03bc-6936-4d42-bf9b-60741aa668ba" (UID: "b00e03bc-6936-4d42-bf9b-60741aa668ba"). InnerVolumeSpecName "kube-api-access-5bbn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:44:16.294942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.294838    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbn7\" (UniqueName: \"kubernetes.io/projected/b00e03bc-6936-4d42-bf9b-60741aa668ba-kube-api-access-5bbn7\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:16.294942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.294885    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:16.294942 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:16.294901    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b00e03bc-6936-4d42-bf9b-60741aa668ba-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:17.005727 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:17.005657    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2"
Nov 03 18:44:17.005727 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:17.005671    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-gwsj2" event={"ID":"b00e03bc-6936-4d42-bf9b-60741aa668ba","Type":"ContainerDied","Data":"ad5588f6711bd3c65b8a3934f98713631f1303653e484944d7f67076bd28c0d0"}
Nov 03 18:44:17.005727 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:17.005719    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5588f6711bd3c65b8a3934f98713631f1303653e484944d7f67076bd28c0d0"
Nov 03 18:44:33.334901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:33.334028    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-kz2rm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:44:33.334901 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:33.334644    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-kz2rm" podUID="cfb9ec24-1401-455e-b146-cd26986e90e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:44:33.464897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:33.460980    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zcfbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:44:33.464897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:33.461047    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zcfbk" podUID="7daacb4d-0cfe-42f6-bb51-6e08c57ddced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:44:33.563123 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:33.563078    4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-6s5qj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:44:33.563440 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:33.563401    4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-6s5qj" podUID="44987738-f4da-4721-93f0-c4abfa00e1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:44:35.165344 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.165310    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"]
Nov 03 18:44:35.165794 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.165650    4493 cpu_manager.go:414] "containerMap: removing container" podUID="b00e03bc-6936-4d42-bf9b-60741aa668ba" containerName="revoke-keys"
Nov 03 18:44:35.165794 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.165661    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00e03bc-6936-4d42-bf9b-60741aa668ba" containerName="revoke-keys"
Nov 03 18:44:35.165794 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.165726    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00e03bc-6936-4d42-bf9b-60741aa668ba" containerName="revoke-keys"
Nov 03 18:44:35.168722 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.168706    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.212716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.212678    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxbd\" (UniqueName: \"kubernetes.io/projected/26353f4e-b0d1-4369-bd3c-b8ee1192074f-kube-api-access-krxbd\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.212716 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.212721    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.212959 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.212749    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.223529 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.223509    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:44:35.223611 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.223590    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:44:35.223806 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.223791    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:44:35.273212 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.273180    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7s4vb\""
Nov 03 18:44:35.274681 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.274658    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"]
Nov 03 18:44:35.313551 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.313508    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.313763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.313644    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxbd\" (UniqueName: \"kubernetes.io/projected/26353f4e-b0d1-4369-bd3c-b8ee1192074f-kube-api-access-krxbd\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.313763 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.313677    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.314275 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.314251    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.316173 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.316154    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.377670 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.377634    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxbd\" (UniqueName: \"kubernetes.io/projected/26353f4e-b0d1-4369-bd3c-b8ee1192074f-kube-api-access-krxbd\") pod \"maas-api-revoke-sub-0165d7d2-vq22s\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.480762 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.480725    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:35.856055 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:35.856027    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"]
Nov 03 18:44:35.858493 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:44:35.858461    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26353f4e_b0d1_4369_bd3c_b8ee1192074f.slice/crio-667521cb11ca5940d4d10c8e51102f4bac6f5c6027ffebed6fa034ef2ef93381 WatchSource:0}: Error finding container 667521cb11ca5940d4d10c8e51102f4bac6f5c6027ffebed6fa034ef2ef93381: Status 404 returned error can't find the container with id 667521cb11ca5940d4d10c8e51102f4bac6f5c6027ffebed6fa034ef2ef93381
Nov 03 18:44:36.100703 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:36.100660    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s" event={"ID":"26353f4e-b0d1-4369-bd3c-b8ee1192074f","Type":"ContainerStarted","Data":"667521cb11ca5940d4d10c8e51102f4bac6f5c6027ffebed6fa034ef2ef93381"}
Nov 03 18:44:36.992109 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:44:36.992072    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:44:37.105947 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:37.105913    4493 generic.go:356] "Generic (PLEG): container finished" podID="26353f4e-b0d1-4369-bd3c-b8ee1192074f" containerID="49064b05553e399ece9fed7cf117db6eaad9d25460e920c364d26eb9684fd7b0" exitCode=0
Nov 03 18:44:37.106211 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:37.105962    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s" event={"ID":"26353f4e-b0d1-4369-bd3c-b8ee1192074f","Type":"ContainerDied","Data":"49064b05553e399ece9fed7cf117db6eaad9d25460e920c364d26eb9684fd7b0"}
Nov 03 18:44:38.237019 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.236990    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:38.341528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.341494    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-xks-ca\") pod \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") "
Nov 03 18:44:38.341528 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.341537    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26353f4e-b0d1-4369-bd3c-b8ee1192074f-kube-api-access-krxbd\" (UniqueName: \"kubernetes.io/projected/26353f4e-b0d1-4369-bd3c-b8ee1192074f-kube-api-access-krxbd\") pod \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") "
Nov 03 18:44:38.341814 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.341579    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-openshift-ca\") pod \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\" (UID: \"26353f4e-b0d1-4369-bd3c-b8ee1192074f\") "
Nov 03 18:44:38.342005 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.341980    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-openshift-ca" pod "26353f4e-b0d1-4369-bd3c-b8ee1192074f" (UID: "26353f4e-b0d1-4369-bd3c-b8ee1192074f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:44:38.343789 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.343769    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-xks-ca" pod "26353f4e-b0d1-4369-bd3c-b8ee1192074f" (UID: "26353f4e-b0d1-4369-bd3c-b8ee1192074f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:44:38.343910 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.343867    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26353f4e-b0d1-4369-bd3c-b8ee1192074f-kube-api-access-krxbd" pod "26353f4e-b0d1-4369-bd3c-b8ee1192074f" (UID: "26353f4e-b0d1-4369-bd3c-b8ee1192074f"). InnerVolumeSpecName "kube-api-access-krxbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:44:38.442315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.442225    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:38.442315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.442256    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krxbd\" (UniqueName: \"kubernetes.io/projected/26353f4e-b0d1-4369-bd3c-b8ee1192074f-kube-api-access-krxbd\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:38.442315 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:38.442268    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26353f4e-b0d1-4369-bd3c-b8ee1192074f-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:39.115409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:39.115333    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s"
Nov 03 18:44:39.115409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:39.115344    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vq22s" event={"ID":"26353f4e-b0d1-4369-bd3c-b8ee1192074f","Type":"ContainerDied","Data":"667521cb11ca5940d4d10c8e51102f4bac6f5c6027ffebed6fa034ef2ef93381"}
Nov 03 18:44:39.115409 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:39.115388    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667521cb11ca5940d4d10c8e51102f4bac6f5c6027ffebed6fa034ef2ef93381"
Nov 03 18:44:47.008229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.008195    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"]
Nov 03 18:44:47.008741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.008523    4493 cpu_manager.go:414] "containerMap: removing container" podUID="26353f4e-b0d1-4369-bd3c-b8ee1192074f" containerName="revoke-keys"
Nov 03 18:44:47.008741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.008534    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26353f4e-b0d1-4369-bd3c-b8ee1192074f" containerName="revoke-keys"
Nov 03 18:44:47.008741 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.008606    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="26353f4e-b0d1-4369-bd3c-b8ee1192074f" containerName="revoke-keys"
Nov 03 18:44:47.011854 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.011831    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.116408 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.116368    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:44:47.116408 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.116398    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7s4vb\""
Nov 03 18:44:47.116659 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.116377    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:44:47.116659 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.116368    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 18:44:47.116771 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.116741    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.116853 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.116783    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntcnd\" (UniqueName: \"kubernetes.io/projected/67222eb3-ba99-4126-92eb-2cca5127ce79-kube-api-access-ntcnd\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.116950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.116926    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.118417 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.118393    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"]
Nov 03 18:44:47.218229 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.218192    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.218412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.218235    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntcnd\" (UniqueName: \"kubernetes.io/projected/67222eb3-ba99-4126-92eb-2cca5127ce79-kube-api-access-ntcnd\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.218412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.218363    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.219022 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.218989    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.220757 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.220738    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.282018 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.281939    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntcnd\" (UniqueName: \"kubernetes.io/projected/67222eb3-ba99-4126-92eb-2cca5127ce79-kube-api-access-ntcnd\") pod \"maas-api-revoke-sub-6b19338f-g9wlz\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.324113 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.324081    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:47.701194 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:47.701167    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"]
Nov 03 18:44:47.703695 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:44:47.703662    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67222eb3_ba99_4126_92eb_2cca5127ce79.slice/crio-3a8ecd0519e30a890a5efc2c1df5be74f6624c546818600d3877c971a2bae1bf WatchSource:0}: Error finding container 3a8ecd0519e30a890a5efc2c1df5be74f6624c546818600d3877c971a2bae1bf: Status 404 returned error can't find the container with id 3a8ecd0519e30a890a5efc2c1df5be74f6624c546818600d3877c971a2bae1bf
Nov 03 18:44:48.154841 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:48.154804    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz" event={"ID":"67222eb3-ba99-4126-92eb-2cca5127ce79","Type":"ContainerStarted","Data":"b8770e579d5a04580d6d5c9d33740b3c3dfeff1475611f56515bcddf8d2a22c9"}
Nov 03 18:44:48.155285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:48.154863    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz" event={"ID":"67222eb3-ba99-4126-92eb-2cca5127ce79","Type":"ContainerStarted","Data":"3a8ecd0519e30a890a5efc2c1df5be74f6624c546818600d3877c971a2bae1bf"}
Nov 03 18:44:48.275816 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:48.275759    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz" podStartSLOduration=2.275747109 podStartE2EDuration="2.275747109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:48.27439866 +0000 UTC m=+4267.733569647" watchObservedRunningTime="2026-11-03 18:44:48.275747109 +0000 UTC m=+4267.734918095"
Nov 03 18:44:49.160772 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:49.160738    4493 generic.go:356] "Generic (PLEG): container finished" podID="67222eb3-ba99-4126-92eb-2cca5127ce79" containerID="b8770e579d5a04580d6d5c9d33740b3c3dfeff1475611f56515bcddf8d2a22c9" exitCode=0
Nov 03 18:44:49.161220 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:49.160780    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz" event={"ID":"67222eb3-ba99-4126-92eb-2cca5127ce79","Type":"ContainerDied","Data":"b8770e579d5a04580d6d5c9d33740b3c3dfeff1475611f56515bcddf8d2a22c9"}
Nov 03 18:44:50.292340 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.292314    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:50.346402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.346364    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67222eb3-ba99-4126-92eb-2cca5127ce79-kube-api-access-ntcnd\" (UniqueName: \"kubernetes.io/projected/67222eb3-ba99-4126-92eb-2cca5127ce79-kube-api-access-ntcnd\") pod \"67222eb3-ba99-4126-92eb-2cca5127ce79\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") "
Nov 03 18:44:50.346578 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.346471    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-xks-ca\") pod \"67222eb3-ba99-4126-92eb-2cca5127ce79\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") "
Nov 03 18:44:50.346578 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.346503    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-openshift-ca\") pod \"67222eb3-ba99-4126-92eb-2cca5127ce79\" (UID: \"67222eb3-ba99-4126-92eb-2cca5127ce79\") "
Nov 03 18:44:50.347004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.346978    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-openshift-ca" pod "67222eb3-ba99-4126-92eb-2cca5127ce79" (UID: "67222eb3-ba99-4126-92eb-2cca5127ce79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:44:50.348779 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.348757    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-xks-ca" pod "67222eb3-ba99-4126-92eb-2cca5127ce79" (UID: "67222eb3-ba99-4126-92eb-2cca5127ce79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:44:50.348915 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.348755    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67222eb3-ba99-4126-92eb-2cca5127ce79-kube-api-access-ntcnd" pod "67222eb3-ba99-4126-92eb-2cca5127ce79" (UID: "67222eb3-ba99-4126-92eb-2cca5127ce79"). InnerVolumeSpecName "kube-api-access-ntcnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:44:50.447808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.447726    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntcnd\" (UniqueName: \"kubernetes.io/projected/67222eb3-ba99-4126-92eb-2cca5127ce79-kube-api-access-ntcnd\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:50.447808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.447758    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-xks-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:50.447808 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:50.447770    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67222eb3-ba99-4126-92eb-2cca5127ce79-maas-api-openshift-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:44:51.171236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:51.171151    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz"
Nov 03 18:44:51.171236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:51.171156    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-g9wlz" event={"ID":"67222eb3-ba99-4126-92eb-2cca5127ce79","Type":"ContainerDied","Data":"3a8ecd0519e30a890a5efc2c1df5be74f6624c546818600d3877c971a2bae1bf"}
Nov 03 18:44:51.171236 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:44:51.171199    4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8ecd0519e30a890a5efc2c1df5be74f6624c546818600d3877c971a2bae1bf"
Nov 03 18:45:00.182644 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.182602    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"]
Nov 03 18:45:00.183062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.183017    4493 cpu_manager.go:414] "containerMap: removing container" podUID="67222eb3-ba99-4126-92eb-2cca5127ce79" containerName="revoke-keys"
Nov 03 18:45:00.183062 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.183030    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67222eb3-ba99-4126-92eb-2cca5127ce79" containerName="revoke-keys"
Nov 03 18:45:00.183137 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.183113    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="67222eb3-ba99-4126-92eb-2cca5127ce79" containerName="revoke-keys"
Nov 03 18:45:00.186128 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.186109    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:00.241025 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.240993    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 18:45:00.290950 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.290911    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 18:45:00.291172 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.290970    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7s4vb\""
Nov 03 18:45:00.292721 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.292690    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"]
Nov 03 18:45:00.355443 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.355399    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-kube-api-access-g9h8v\") pod \"maas-api-key-cleanup-29895525-ll74c\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:00.355718 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.355693    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895525-ll74c\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:00.457455 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.457336    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-kube-api-access-g9h8v\") pod \"maas-api-key-cleanup-29895525-ll74c\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:00.458175 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.458145    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895525-ll74c\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:00.459334 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.459125    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895525-ll74c\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:00.563906 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.545446    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-kube-api-access-g9h8v\") pod \"maas-api-key-cleanup-29895525-ll74c\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:00.799073 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:00.799035    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:01.277897 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:01.277791    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"]
Nov 03 18:45:01.304710 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:45:01.304677    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aebc6c0_7a70_4d6c_bf89_bb22db4b036d.slice/crio-ae5a2a96a66473b37924b1a60280253f21ffc4ccc220f2b8035db7abff19aabd WatchSource:0}: Error finding container ae5a2a96a66473b37924b1a60280253f21ffc4ccc220f2b8035db7abff19aabd: Status 404 returned error can't find the container with id ae5a2a96a66473b37924b1a60280253f21ffc4ccc220f2b8035db7abff19aabd
Nov 03 18:45:02.216704 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:02.216665    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" event={"ID":"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d","Type":"ContainerStarted","Data":"893b645d3c58b64896d922ab34f6fe87c38cd55993c97f55f6d0f2f3a63d0ee9"}
Nov 03 18:45:02.216909 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:02.216722    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" event={"ID":"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d","Type":"ContainerStarted","Data":"ae5a2a96a66473b37924b1a60280253f21ffc4ccc220f2b8035db7abff19aabd"}
Nov 03 18:45:02.335933 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:02.335863    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" podStartSLOduration=2.335847967 podStartE2EDuration="2.335847967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:02.335058358 +0000 UTC m=+4281.794229344" watchObservedRunningTime="2026-11-03 18:45:02.335847967 +0000 UTC m=+4281.795019005"
Nov 03 18:45:03.222404 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:03.222374    4493 generic.go:356] "Generic (PLEG): container finished" podID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerID="893b645d3c58b64896d922ab34f6fe87c38cd55993c97f55f6d0f2f3a63d0ee9" exitCode=60
Nov 03 18:45:03.222586 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:03.222464    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" event={"ID":"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d","Type":"ContainerDied","Data":"893b645d3c58b64896d922ab34f6fe87c38cd55993c97f55f6d0f2f3a63d0ee9"}
Nov 03 18:45:03.222677 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:03.222665    4493 scope.go:122] "RemoveContainer" containerID="893b645d3c58b64896d922ab34f6fe87c38cd55993c97f55f6d0f2f3a63d0ee9"
Nov 03 18:45:04.228943 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:04.228909    4493 generic.go:356] "Generic (PLEG): container finished" podID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerID="e28e5d0b9155836ff2e4caab68a31e3f5101a7e8bb1c3710a94e391243b82891" exitCode=60
Nov 03 18:45:04.229436 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:04.229010    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" event={"ID":"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d","Type":"ContainerDied","Data":"e28e5d0b9155836ff2e4caab68a31e3f5101a7e8bb1c3710a94e391243b82891"}
Nov 03 18:45:04.229436 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:04.229086    4493 scope.go:122] "RemoveContainer" containerID="893b645d3c58b64896d922ab34f6fe87c38cd55993c97f55f6d0f2f3a63d0ee9"
Nov 03 18:45:04.229436 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:04.229354    4493 scope.go:122] "RemoveContainer" containerID="e28e5d0b9155836ff2e4caab68a31e3f5101a7e8bb1c3710a94e391243b82891"
Nov 03 18:45:04.229629 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:45:04.229607    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895525-ll74c_odh-ai-gateway-infra(3aebc6c0-7a70-4d6c-bf89-bb22db4b036d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d"
Nov 03 18:45:05.235266 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:05.235239    4493 scope.go:122] "RemoveContainer" containerID="e28e5d0b9155836ff2e4caab68a31e3f5101a7e8bb1c3710a94e391243b82891"
Nov 03 18:45:05.235667 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:45:05.235457    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895525-ll74c_odh-ai-gateway-infra(3aebc6c0-7a70-4d6c-bf89-bb22db4b036d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d"
Nov 03 18:45:06.990613 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:45:06.990568    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:45:11.403262 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:11.403228    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dbx77_ab184125-d586-48cc-944c-8ec77fb048e0/manager/0.log"
Nov 03 18:45:12.718311 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:12.718280    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mr6xh_7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6/proxy/0.log"
Nov 03 18:45:14.073894 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:14.073843    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29895525-ll74c_3aebc6c0-7a70-4d6c-bf89-bb22db4b036d/cleanup/1.log"
Nov 03 18:45:14.284437 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:14.284403    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vq22s_26353f4e-b0d1-4369-bd3c-b8ee1192074f/revoke-keys/0.log"
Nov 03 18:45:14.509414 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:14.509384    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-g9wlz_67222eb3-ba99-4126-92eb-2cca5127ce79/revoke-keys/0.log"
Nov 03 18:45:14.726890 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:14.726843    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-gwsj2_b00e03bc-6936-4d42-bf9b-60741aa668ba/revoke-keys/0.log"
Nov 03 18:45:15.425752 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:15.425726    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29895525-ll74c_3aebc6c0-7a70-4d6c-bf89-bb22db4b036d/cleanup/1.log"
Nov 03 18:45:15.705502 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:15.705436    4493 scope.go:122] "RemoveContainer" containerID="e28e5d0b9155836ff2e4caab68a31e3f5101a7e8bb1c3710a94e391243b82891"
Nov 03 18:45:16.282243 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:16.282156    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" event={"ID":"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d","Type":"ContainerStarted","Data":"6b4423ee3ef1441fab3c85864fb4482e4f3345e16bc3348420e17db149a6893f"}
Nov 03 18:45:17.289004 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:17.288974    4493 generic.go:356] "Generic (PLEG): container finished" podID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerID="6b4423ee3ef1441fab3c85864fb4482e4f3345e16bc3348420e17db149a6893f" exitCode=60
Nov 03 18:45:17.289475 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:17.289046    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" event={"ID":"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d","Type":"ContainerDied","Data":"6b4423ee3ef1441fab3c85864fb4482e4f3345e16bc3348420e17db149a6893f"}
Nov 03 18:45:17.289475 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:17.289119    4493 scope.go:122] "RemoveContainer" containerID="e28e5d0b9155836ff2e4caab68a31e3f5101a7e8bb1c3710a94e391243b82891"
Nov 03 18:45:17.289475 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:17.289471    4493 scope.go:122] "RemoveContainer" containerID="6b4423ee3ef1441fab3c85864fb4482e4f3345e16bc3348420e17db149a6893f"
Nov 03 18:45:17.289740 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:45:17.289723    4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895525-ll74c_odh-ai-gateway-infra(3aebc6c0-7a70-4d6c-bf89-bb22db4b036d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d"
Nov 03 18:45:17.425156 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:17.425122    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"]
Nov 03 18:45:18.423445 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:18.423419    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:18.558464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:18.558380    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-kube-api-access-g9h8v\") pod \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") "
Nov 03 18:45:18.558464 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:18.558438    4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-maas-api-service-ca\") pod \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\" (UID: \"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d\") "
Nov 03 18:45:18.558955 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:18.558925    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-maas-api-service-ca" pod "3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" (UID: "3aebc6c0-7a70-4d6c-bf89-bb22db4b036d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:18.560723 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:18.560691    4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-kube-api-access-g9h8v" pod "3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" (UID: "3aebc6c0-7a70-4d6c-bf89-bb22db4b036d"). InnerVolumeSpecName "kube-api-access-g9h8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:45:18.659981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:18.659933    4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9h8v\" (UniqueName: \"kubernetes.io/projected/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-kube-api-access-g9h8v\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:45:18.659981 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:18.659974    4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d-maas-api-service-ca\") on node \"ip-10-0-131-172.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 18:45:19.300449 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:19.300415    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c" event={"ID":"3aebc6c0-7a70-4d6c-bf89-bb22db4b036d","Type":"ContainerDied","Data":"ae5a2a96a66473b37924b1a60280253f21ffc4ccc220f2b8035db7abff19aabd"}
Nov 03 18:45:19.300449 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:19.300450    4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"
Nov 03 18:45:19.300697 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:19.300477    4493 scope.go:122] "RemoveContainer" containerID="6b4423ee3ef1441fab3c85864fb4482e4f3345e16bc3348420e17db149a6893f"
Nov 03 18:45:19.469662 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:19.469625    4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"]
Nov 03 18:45:19.520446 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:19.520409    4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895525-ll74c"]
Nov 03 18:45:20.155537 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:20.155499    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8f76b86f-6w6hr_1e569d24-754e-4f20-9448-29cb410869d6/kube-auth-proxy/0.log"
Nov 03 18:45:20.712159 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:20.712128    4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" path="/var/lib/kubelet/pods/3aebc6c0-7a70-4d6c-bf89-bb22db4b036d/volumes"
Nov 03 18:45:20.919136 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:20.919106    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-6s5qj_44987738-f4da-4721-93f0-c4abfa00e1b1/istio-proxy/0.log"
Nov 03 18:45:21.927456 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:21.927428    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-kz2rm_cfb9ec24-1401-455e-b146-cd26986e90e1/istio-proxy/0.log"
Nov 03 18:45:23.068862 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:23.068827    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-zcfbk_7daacb4d-0cfe-42f6-bb51-6e08c57ddced/istio-proxy/0.log"
Nov 03 18:45:36.991625 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:45:36.991512    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:45:38.175100 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:38.175059    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-dbx77_ab184125-d586-48cc-944c-8ec77fb048e0/manager/0.log"
Nov 03 18:45:39.374499 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:39.374470    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-mr6xh_7d2f5f6b-99b5-4be6-a9f1-98b05f468ae6/proxy/0.log"
Nov 03 18:45:40.644935 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:40.644896    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vq22s_26353f4e-b0d1-4369-bd3c-b8ee1192074f/revoke-keys/0.log"
Nov 03 18:45:40.868045 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:40.868017    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-g9wlz_67222eb3-ba99-4126-92eb-2cca5127ce79/revoke-keys/0.log"
Nov 03 18:45:41.092290 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:41.092259    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-gwsj2_b00e03bc-6936-4d42-bf9b-60741aa668ba/revoke-keys/0.log"
Nov 03 18:45:45.578402 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:45.578374    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8f76b86f-6w6hr_1e569d24-754e-4f20-9448-29cb410869d6/kube-auth-proxy/0.log"
Nov 03 18:45:46.310378 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:46.310347    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-6s5qj_44987738-f4da-4721-93f0-c4abfa00e1b1/istio-proxy/0.log"
Nov 03 18:45:47.327622 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:47.327587    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-kz2rm_cfb9ec24-1401-455e-b146-cd26986e90e1/istio-proxy/0.log"
Nov 03 18:45:48.404259 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:45:48.404227    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-zcfbk_7daacb4d-0cfe-42f6-bb51-6e08c57ddced/istio-proxy/0.log"
Nov 03 18:46:06.991127 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:46:06.991091    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:46:36.991205 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:46:36.991165    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:47:06.992113 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:47:06.992068    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:47:16.087743 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:16.087711    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-648bg_9b82e04a-d3d2-43de-ad90-abccdeb48e7e/global-pull-secret-syncer/0.log"
Nov 03 18:47:19.727382 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:19.727351    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t4sns_c1a0388c-1177-45a8-8eb8-92ef23d8f0bf/konnectivity-agent/0.log"
Nov 03 18:47:20.633834 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:20.633800    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-172.us-west-1.compute.internal_3eefabd2e362fe1b3609803d32f993c7/haproxy/0.log"
Nov 03 18:47:26.379748 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.379712    4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"]
Nov 03 18:47:26.380147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380052    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.380147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380063    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.380147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380083    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.380147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380088    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.380147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380094    4493 cpu_manager.go:414] "containerMap: removing container" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.380147 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380099    4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.380414 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380152    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.380414 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.380160    4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aebc6c0-7a70-4d6c-bf89-bb22db4b036d" containerName="cleanup"
Nov 03 18:47:26.383424 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.383403    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.438278 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.438250    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlnz7\"/\"openshift-service-ca.crt\""
Nov 03 18:47:26.488097 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.488070    4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vlnz7\"/\"default-dockercfg-ngffq\""
Nov 03 18:47:26.488253 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.488126    4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlnz7\"/\"kube-root-ca.crt\""
Nov 03 18:47:26.489654 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.489632    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"]
Nov 03 18:47:26.490100 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.490072    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-podres\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.490233 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.490219    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-lib-modules\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.490303 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.490283    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtr2\" (UniqueName: \"kubernetes.io/projected/626bf816-a643-40f4-981d-de6d3fdbed94-kube-api-access-gbtr2\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.490352 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.490319    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-proc\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.490419 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.490400    4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-sys\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591011 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.590979    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-sys\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591027    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-podres\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591101    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-lib-modules\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591132    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtr2\" (UniqueName: \"kubernetes.io/projected/626bf816-a643-40f4-981d-de6d3fdbed94-kube-api-access-gbtr2\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591139    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-podres\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591191 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591156    4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-proc\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591389 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591198    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-lib-modules\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591389 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591200    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-proc\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.591389 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.591101    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/626bf816-a643-40f4-981d-de6d3fdbed94-sys\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.655285 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.655221    4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtr2\" (UniqueName: \"kubernetes.io/projected/626bf816-a643-40f4-981d-de6d3fdbed94-kube-api-access-gbtr2\") pod \"perf-node-gather-daemonset-g7shp\" (UID: \"626bf816-a643-40f4-981d-de6d3fdbed94\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:26.695468 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:26.695439    4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:27.075295 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:27.075264    4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"]
Nov 03 18:47:27.077792 ip-10-0-131-172 kubenswrapper[4493]: W1103 18:47:27.077757    4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod626bf816_a643_40f4_981d_de6d3fdbed94.slice/crio-ae95d2a9cf22d436eb7fc4efee383d5f1083a8e328b80a0fd4b249fe04f29f1f WatchSource:0}: Error finding container ae95d2a9cf22d436eb7fc4efee383d5f1083a8e328b80a0fd4b249fe04f29f1f: Status 404 returned error can't find the container with id ae95d2a9cf22d436eb7fc4efee383d5f1083a8e328b80a0fd4b249fe04f29f1f
Nov 03 18:47:27.849812 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:27.849758    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp" event={"ID":"626bf816-a643-40f4-981d-de6d3fdbed94","Type":"ContainerStarted","Data":"f04b673ba3cc99a27e058d2d2d05d74130daab9838cadc813d11920d9b524445"}
Nov 03 18:47:27.849812 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:27.849819    4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp" event={"ID":"626bf816-a643-40f4-981d-de6d3fdbed94","Type":"ContainerStarted","Data":"ae95d2a9cf22d436eb7fc4efee383d5f1083a8e328b80a0fd4b249fe04f29f1f"}
Nov 03 18:47:27.850412 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:27.849900    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:27.969780 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:27.969722    4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp" podStartSLOduration=1.969709211 podStartE2EDuration="1.969709211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:27.969140423 +0000 UTC m=+4427.428311410" watchObservedRunningTime="2026-11-03 18:47:27.969709211 +0000 UTC m=+4427.428880191"
Nov 03 18:47:33.865465 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:33.865429    4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-g7shp"
Nov 03 18:47:36.240774 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:36.240745    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64477db88b-vv6ll_a9d82d34-457f-4bbd-b196-9047c48ee579/metrics-server/0.log"
Nov 03 18:47:36.999284 ip-10-0-131-172 kubenswrapper[4493]: E1103 18:47:36.999249    4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-82x2r" containerName="extractor"
Nov 03 18:47:39.644700 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:39.644657    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qss6d_b5b8188d-5925-4eca-af95-23c3efd5a999/node-exporter/0.log"
Nov 03 18:47:40.064223 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:40.064186    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qss6d_b5b8188d-5925-4eca-af95-23c3efd5a999/kube-rbac-proxy/0.log"
Nov 03 18:47:40.491977 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:40.491951    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qss6d_b5b8188d-5925-4eca-af95-23c3efd5a999/init-textfile/0.log"
Nov 03 18:47:42.197363 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:42.197332    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a7b2c5-bb90-42d6-8529-0a9d4444d356/prometheus/0.log"
Nov 03 18:47:42.618413 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:42.618385    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a7b2c5-bb90-42d6-8529-0a9d4444d356/config-reloader/0.log"
Nov 03 18:47:43.045813 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:43.045776    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a7b2c5-bb90-42d6-8529-0a9d4444d356/thanos-sidecar/0.log"
Nov 03 18:47:43.469495 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:43.469466    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a7b2c5-bb90-42d6-8529-0a9d4444d356/kube-rbac-proxy-web/0.log"
Nov 03 18:47:43.842111 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:43.842020    4493 scope.go:122] "RemoveContainer" containerID="4e1714c28187ebbb6e7e6b4cfd978739398e179426de7203bd74409af51bb272"
Nov 03 18:47:43.891917 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:43.891864    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a7b2c5-bb90-42d6-8529-0a9d4444d356/kube-rbac-proxy/0.log"
Nov 03 18:47:44.314352 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:44.314322    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a7b2c5-bb90-42d6-8529-0a9d4444d356/kube-rbac-proxy-thanos/0.log"
Nov 03 18:47:44.736958 ip-10-0-131-172 kubenswrapper[4493]: I1103 18:47:44.736924    4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07a7b2c5-bb90-42d6-8529-0a9d4444d356/init-config-reloader/0.log"